[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:15:13 dist-test-slave-nbjg krb5kdc[14273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:15:13 dist-test-slave-nbjg krb5kdc[14273](info): set up 2 sockets
Sep 29 03:15:13 dist-test-slave-nbjg krb5kdc[14273](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 03:15:14.530539 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:15:14 dist-test-slave-nbjg krb5kdc[14273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:15:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:15:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:15:14.650182 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:44667
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:44667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:14.759294 14286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:14.759557 14286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:14.759598 14286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:14.763542 14286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:14.763603 14286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:14.763622 14286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:14.763638 14286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:14.763652 14286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:15:14.769125 14286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:44667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:44667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:14.770356 14286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:14.771373 14286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:14.780361 14291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:14.780472 14286 server_base.cc:1034] running on GCE node
W20260929 03:15:14.780357 14292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:14.780341 14294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:14.781240 14286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:14.782588 14286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:14.783769 14286 hybrid_clock.cc:648] HybridClock initialized: now 1790651714783744 us; error 48 us; skew 500 ppm
Sep 29 03:15:14 dist-test-slave-nbjg krb5kdc[14273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651714, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:14.786592 14286 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:15:14.787513 14286 webserver.cc:460] Webserver started at http://127.13.238.190:43611/ using document root <none> and password file <none>
I20260929 03:15:14.788066 14286 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:14.788122 14286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:14.788287 14286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:14.790573 14286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "b9531b9dfaf14008a4a4e7c9d2a9346e"
format_stamp: "Formatted at 2026-09-29 03:15:14 on dist-test-slave-nbjg"
server_key: "47d86b44aa4b7417d19904db02542c10"
server_key_iv: "324f1229f1946f62587659beea6727ed"
server_key_version: "encryptionkey@0"
I20260929 03:15:14.791141 14286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "b9531b9dfaf14008a4a4e7c9d2a9346e"
format_stamp: "Formatted at 2026-09-29 03:15:14 on dist-test-slave-nbjg"
server_key: "47d86b44aa4b7417d19904db02542c10"
server_key_iv: "324f1229f1946f62587659beea6727ed"
server_key_version: "encryptionkey@0"
I20260929 03:15:14.795222 14286 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 03:15:14.797889 14301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:14.799299 14286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:15:14.799436 14286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "b9531b9dfaf14008a4a4e7c9d2a9346e"
format_stamp: "Formatted at 2026-09-29 03:15:14 on dist-test-slave-nbjg"
server_key: "47d86b44aa4b7417d19904db02542c10"
server_key_iv: "324f1229f1946f62587659beea6727ed"
server_key_version: "encryptionkey@0"
I20260929 03:15:14.799561 14286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:14.814858 14286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:14.818177 14286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:14.818374 14286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:14.825335 14286 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:44667
I20260929 03:15:14.825591 14353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:44667 every 8 connection(s)
I20260929 03:15:14.826347 14286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:15:14.829276 14354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:14.834805 14354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9531b9dfaf14008a4a4e7c9d2a9346e: Bootstrap starting.
I20260929 03:15:14.836491 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 14286
I20260929 03:15:14.836627 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:15:14.837332 14354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9531b9dfaf14008a4a4e7c9d2a9346e: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:14.837913 14266 external_mini_cluster.cc:1428] Setting key 6df2416e80615e3dfbb32ef1287e063a
I20260929 03:15:14.838223 14354 log.cc:824] T 00000000000000000000000000000000 P b9531b9dfaf14008a4a4e7c9d2a9346e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:14.840219 14354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9531b9dfaf14008a4a4e7c9d2a9346e: No bootstrap required, opened a new log
I20260929 03:15:14.844388 14354 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9531b9dfaf14008a4a4e7c9d2a9346e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9531b9dfaf14008a4a4e7c9d2a9346e" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 44667 } }
I20260929 03:15:14.844621 14354 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9531b9dfaf14008a4a4e7c9d2a9346e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:14.844686 14354 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9531b9dfaf14008a4a4e7c9d2a9346e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9531b9dfaf14008a4a4e7c9d2a9346e, State: Initialized, Role: FOLLOWER
I20260929 03:15:14.845158 14354 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9531b9dfaf14008a4a4e7c9d2a9346e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9531b9dfaf14008a4a4e7c9d2a9346e" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 44667 } }
I20260929 03:15:14.845312 14354 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9531b9dfaf14008a4a4e7c9d2a9346e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:15:14.845376 14354 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9531b9dfaf14008a4a4e7c9d2a9346e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:15:14.845467 14354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9531b9dfaf14008a4a4e7c9d2a9346e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:14.846611 14354 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9531b9dfaf14008a4a4e7c9d2a9346e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9531b9dfaf14008a4a4e7c9d2a9346e" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 44667 } }
I20260929 03:15:14.846936 14354 leader_election.cc:302] T 00000000000000000000000000000000 P b9531b9dfaf14008a4a4e7c9d2a9346e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9531b9dfaf14008a4a4e7c9d2a9346e; no voters:
I20260929 03:15:14.847214 14354 leader_election.cc:288] T 00000000000000000000000000000000 P b9531b9dfaf14008a4a4e7c9d2a9346e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:15:14.847316 14359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9531b9dfaf14008a4a4e7c9d2a9346e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:14.847532 14359 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9531b9dfaf14008a4a4e7c9d2a9346e [term 1 LEADER]: Becoming Leader. State: Replica: b9531b9dfaf14008a4a4e7c9d2a9346e, State: Running, Role: LEADER
I20260929 03:15:14.847841 14359 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9531b9dfaf14008a4a4e7c9d2a9346e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9531b9dfaf14008a4a4e7c9d2a9346e" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 44667 } }
I20260929 03:15:14.848368 14354 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9531b9dfaf14008a4a4e7c9d2a9346e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:14.849090 14361 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9531b9dfaf14008a4a4e7c9d2a9346e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9531b9dfaf14008a4a4e7c9d2a9346e. Latest consensus state: current_term: 1 leader_uuid: "b9531b9dfaf14008a4a4e7c9d2a9346e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9531b9dfaf14008a4a4e7c9d2a9346e" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 44667 } } }
I20260929 03:15:14.849221 14361 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9531b9dfaf14008a4a4e7c9d2a9346e [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:14.849522 14360 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9531b9dfaf14008a4a4e7c9d2a9346e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9531b9dfaf14008a4a4e7c9d2a9346e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9531b9dfaf14008a4a4e7c9d2a9346e" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 44667 } } }
I20260929 03:15:14.849637 14360 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9531b9dfaf14008a4a4e7c9d2a9346e [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:14.849900 14368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:15:14.853168 14368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 03:15:14 dist-test-slave-nbjg krb5kdc[14273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:14.862519 14368 catalog_manager.cc:1347] Generated new cluster ID: 272675f0fffa4a83a8bc1fdaf0d60fa6
I20260929 03:15:14.862584 14368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:15:14.864569 14357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:14.840879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45924 (local address 127.13.238.190:44667)
0929 03:15:14.841384 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:14.841402 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:14.842605 (+ 1203us) server_negotiation.cc:407] Connection header received
0929 03:15:14.845166 (+ 2561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:14.845185 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:14.845596 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:14.845958 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:14.848507 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:14.850115 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:14.851895 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:14.852369 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:14.852524 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:14.857288 (+ 4764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:14.857294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:14.857311 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:14.857342 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:14.860237 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:14.860884 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:14.860890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:14.860895 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:14.861047 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:14.861442 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:14.861446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:14.861448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:14.861956 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:14.862885 (+ 929us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:14.863053 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 03:15:14.863686 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":123,"threads_started":1}
I20260929 03:15:14.874946 14368 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:15:14.875793 14368 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:15:14.880666 14368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9531b9dfaf14008a4a4e7c9d2a9346e: Generated new TSK 0
I20260929 03:15:14.881313 14368 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
I20260929 03:15:14.909353 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:0
--local_ip_for_outbound_sockets=127.13.238.129
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:44667
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:15.020673 14380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:15.020961 14380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:15.021008 14380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:15.025166 14380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:15.025233 14380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:15.025295 14380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:15:15.030612 14380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:44667
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:15.031749 14380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:15.032824 14380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:15.042429 14386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:15.042420 14385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:15.042420 14388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:15.042524 14380 server_base.cc:1034] running on GCE node
I20260929 03:15:15.043131 14380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:15.043731 14380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:15.044925 14380 hybrid_clock.cc:648] HybridClock initialized: now 1790651715044905 us; error 30 us; skew 500 ppm
Sep 29 03:15:15 dist-test-slave-nbjg krb5kdc[14273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651715, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:15.047580 14380 init.cc:377] Logged in from keytab as kudu/127.13.238.129@KRBTEST.COM (short username kudu)
I20260929 03:15:15.048478 14380 webserver.cc:460] Webserver started at http://127.13.238.129:41155/ using document root <none> and password file <none>
I20260929 03:15:15.049031 14380 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:15.049103 14380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:15.049341 14380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:15.051386 14380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-0/data/instance:
uuid: "138c11bd9f5e49e89b7748588168564b"
format_stamp: "Formatted at 2026-09-29 03:15:15 on dist-test-slave-nbjg"
server_key: "ee0cc7202cf23e81fdc56d9b1784ab5e"
server_key_iv: "3339d208fb64581e6ab6716631728871"
server_key_version: "encryptionkey@0"
I20260929 03:15:15.051980 14380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance:
uuid: "138c11bd9f5e49e89b7748588168564b"
format_stamp: "Formatted at 2026-09-29 03:15:15 on dist-test-slave-nbjg"
server_key: "ee0cc7202cf23e81fdc56d9b1784ab5e"
server_key_iv: "3339d208fb64581e6ab6716631728871"
server_key_version: "encryptionkey@0"
I20260929 03:15:15.055611 14380 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:15:15.057972 14395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:15.059270 14380 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 03:15:15.059401 14380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "138c11bd9f5e49e89b7748588168564b"
format_stamp: "Formatted at 2026-09-29 03:15:15 on dist-test-slave-nbjg"
server_key: "ee0cc7202cf23e81fdc56d9b1784ab5e"
server_key_iv: "3339d208fb64581e6ab6716631728871"
server_key_version: "encryptionkey@0"
I20260929 03:15:15.059504 14380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:15.080085 14380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:15.083237 14380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:15.083422 14380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:15.084013 14380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:15.084836 14380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:15.084882 14380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:15.084923 14380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:15.084951 14380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:15.093442 14380 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:42909
I20260929 03:15:15.093482 14508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:42909 every 8 connection(s)
I20260929 03:15:15.094446 14380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:15:15.095718 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 14380
I20260929 03:15:15.095820 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance
I20260929 03:15:15.096063 14266 external_mini_cluster.cc:1428] Setting key c426ed0a06d814abd7ef47b13dae8174
Sep 29 03:15:15 dist-test-slave-nbjg krb5kdc[14273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651715, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.238.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.130@KRBTEST.COM" created.
I20260929 03:15:15.108986 14357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:15.096409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:37263 (local address 127.13.238.190:44667)
0929 03:15:15.096564 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:15.096569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:15.097415 (+ 846us) server_negotiation.cc:407] Connection header received
0929 03:15:15.098814 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:15.098818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:15.098881 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:15.098967 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:15.100427 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:15.100951 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:15.102087 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:15.102360 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:15.102452 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:15.105255 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:15.105260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:15.105262 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:15.105297 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:15.106691 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:15.107259 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:15.107263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:15.107264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:15.107347 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:15.107759 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:15.107763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:15.107765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:15.107992 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:15.108081 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:15.108716 (+ 635us) server_negotiation.cc:299] Negotiation successful
0929 03:15:15.108839 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 03:15:15.110018 14511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:15.096798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44667 (local address 127.13.238.129:37263)
0929 03:15:15.097265 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:15.097318 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:15.098614 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:15.099166 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:15.099175 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:15.099649 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:15.100285 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:15.100301 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:15.101074 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:15.101077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:15.101964 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:15.101973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:15.102504 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:15.102508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:15.102707 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:15.103310 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:15.103331 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:15.105025 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:15.106814 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:15.106821 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:15.106842 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:15.107155 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:15.107437 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:15.107441 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:15.107443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:15.107631 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:15.108111 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:15.108118 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:15.108465 (+ 347us) client_negotiation.cc:769] Sending connection context
0929 03:15:15.108681 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 03:15:15.109059 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":101,"threads_started":1}
I20260929 03:15:15.111366 14509 heartbeater.cc:347] Connected to a master server at 127.13.238.190:44667
I20260929 03:15:15.111624 14509 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:15.112136 14509 heartbeater.cc:511] Master 127.13.238.190:44667 requested a full tablet report, sending...
I20260929 03:15:15.113968 14318 ts_manager.cc:194] Registered new tserver with Master: 138c11bd9f5e49e89b7748588168564b (127.13.238.129:42909)
I20260929 03:15:15.115746 14318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.129@KRBTEST.COM'} at 127.13.238.129:37263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
I20260929 03:15:15.124778 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:0
--local_ip_for_outbound_sockets=127.13.238.130
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:44667
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:15.234879 14514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:15.235173 14514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:15.235224 14514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:15.239398 14514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:15.239466 14514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:15.239538 14514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:15:15.244964 14514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:44667
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:15.246039 14514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:15.247109 14514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:15.256614 14519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:15.256651 14520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:15.256655 14522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:15.256855 14514 server_base.cc:1034] running on GCE node
I20260929 03:15:15.257251 14514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:15.257932 14514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:15.259128 14514 hybrid_clock.cc:648] HybridClock initialized: now 1790651715259084 us; error 57 us; skew 500 ppm
Sep 29 03:15:15 dist-test-slave-nbjg krb5kdc[14273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651715, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:15.261831 14514 init.cc:377] Logged in from keytab as kudu/127.13.238.130@KRBTEST.COM (short username kudu)
I20260929 03:15:15.262828 14514 webserver.cc:460] Webserver started at http://127.13.238.130:38853/ using document root <none> and password file <none>
I20260929 03:15:15.263433 14514 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:15.263504 14514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:15.263742 14514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:15.265929 14514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-1/data/instance:
uuid: "51e2d987c9c74e2692f91eda1af2fa6d"
format_stamp: "Formatted at 2026-09-29 03:15:15 on dist-test-slave-nbjg"
server_key: "8b3e744cbbe6124813263915433a1f52"
server_key_iv: "3a2d59809cb02ad52a479e542428c2ff"
server_key_version: "encryptionkey@0"
I20260929 03:15:15.266594 14514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance:
uuid: "51e2d987c9c74e2692f91eda1af2fa6d"
format_stamp: "Formatted at 2026-09-29 03:15:15 on dist-test-slave-nbjg"
server_key: "8b3e744cbbe6124813263915433a1f52"
server_key_iv: "3a2d59809cb02ad52a479e542428c2ff"
server_key_version: "encryptionkey@0"
I20260929 03:15:15.270790 14514 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:15:15.273236 14529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:15.274549 14514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:15.274655 14514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "51e2d987c9c74e2692f91eda1af2fa6d"
format_stamp: "Formatted at 2026-09-29 03:15:15 on dist-test-slave-nbjg"
server_key: "8b3e744cbbe6124813263915433a1f52"
server_key_iv: "3a2d59809cb02ad52a479e542428c2ff"
server_key_version: "encryptionkey@0"
I20260929 03:15:15.274765 14514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:15.292537 14514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:15.295374 14514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:15.295540 14514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:15.296120 14514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:15.296908 14514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:15.296952 14514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:15.296993 14514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:15.297020 14514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:15.305778 14514 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:37305
I20260929 03:15:15.305804 14642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:37305 every 8 connection(s)
I20260929 03:15:15.306699 14514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:15:15.310694 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 14514
I20260929 03:15:15.310798 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance
I20260929 03:15:15.311091 14266 external_mini_cluster.cc:1428] Setting key a1145e6691cc3862390c133f69103578
Sep 29 03:15:15 dist-test-slave-nbjg krb5kdc[14273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651715, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.238.131@KRBTEST.COM; defaulting to no policy
I20260929 03:15:15.320241 14357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:15.308509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:44117 (local address 127.13.238.190:44667)
0929 03:15:15.308658 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:15.308662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:15.309507 (+ 845us) server_negotiation.cc:407] Connection header received
0929 03:15:15.310910 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:15.310913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:15.310994 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:15.311080 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:15.312443 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:15.312959 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:15.314064 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:15.314331 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:15.314428 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:15.316943 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:15.316946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:15.316948 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:15.316972 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:15.318300 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:15.318761 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:15.318764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:15.318766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:15.318837 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:15.319150 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:15.319153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:15.319155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:15.319328 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:15.319412 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:15.319946 (+ 534us) server_negotiation.cc:299] Negotiation successful
0929 03:15:15.320074 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 03:15:15.321061 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:15.308899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44667 (local address 127.13.238.130:44117)
0929 03:15:15.309345 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:15.309393 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:15.310683 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:15.311271 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:15.311279 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:15.311640 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:15.312300 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:15.312320 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:15.313076 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:15.313079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:15.313947 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:15.313956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:15.314477 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:15.314481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:15.314638 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:15.315205 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:15.315221 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:15.316781 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:15.318404 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:15.318410 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:15.318427 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:15.318666 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:15.318926 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:15.318929 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:15.318931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:15.319065 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:15.319424 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:15.319431 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:15.319708 (+ 277us) client_negotiation.cc:769] Sending connection context
0929 03:15:15.319881 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 03:15:15.320196 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.131@KRBTEST.COM" created.
I20260929 03:15:15.322243 14643 heartbeater.cc:347] Connected to a master server at 127.13.238.190:44667
I20260929 03:15:15.322492 14643 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:15.322968 14643 heartbeater.cc:511] Master 127.13.238.190:44667 requested a full tablet report, sending...
I20260929 03:15:15.324018 14318 ts_manager.cc:194] Registered new tserver with Master: 51e2d987c9c74e2692f91eda1af2fa6d (127.13.238.130:37305)
I20260929 03:15:15.324565 14318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.130@KRBTEST.COM'} at 127.13.238.130:44117
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
I20260929 03:15:15.336627 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:0
--local_ip_for_outbound_sockets=127.13.238.131
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:44667
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:15.444849 14648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:15.445106 14648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:15.445147 14648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:15.449220 14648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:15.449287 14648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:15.449349 14648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:15:15.454928 14648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:44667
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:15.455997 14648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:15.457041 14648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:15.466701 14653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:15.466701 14654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:15.466723 14656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:15.466957 14648 server_base.cc:1034] running on GCE node
I20260929 03:15:15.467442 14648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:15.468102 14648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:15.469285 14648 hybrid_clock.cc:648] HybridClock initialized: now 1790651715469263 us; error 28 us; skew 500 ppm
Sep 29 03:15:15 dist-test-slave-nbjg krb5kdc[14273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651715, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:15.472115 14648 init.cc:377] Logged in from keytab as kudu/127.13.238.131@KRBTEST.COM (short username kudu)
I20260929 03:15:15.473021 14648 webserver.cc:460] Webserver started at http://127.13.238.131:39561/ using document root <none> and password file <none>
I20260929 03:15:15.473598 14648 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:15.473661 14648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:15.473858 14648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:15.476046 14648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-2/data/instance:
uuid: "fe8818986f7e467aa1074954df3f0201"
format_stamp: "Formatted at 2026-09-29 03:15:15 on dist-test-slave-nbjg"
server_key: "5555d4767699ec1752a57ec06059900f"
server_key_iv: "a9a1725aa2d7f41c227f4a6e3c9b0707"
server_key_version: "encryptionkey@0"
I20260929 03:15:15.476593 14648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance:
uuid: "fe8818986f7e467aa1074954df3f0201"
format_stamp: "Formatted at 2026-09-29 03:15:15 on dist-test-slave-nbjg"
server_key: "5555d4767699ec1752a57ec06059900f"
server_key_iv: "a9a1725aa2d7f41c227f4a6e3c9b0707"
server_key_version: "encryptionkey@0"
I20260929 03:15:15.480566 14648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:15:15.483075 14663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:15.484339 14648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:15.484467 14648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "fe8818986f7e467aa1074954df3f0201"
format_stamp: "Formatted at 2026-09-29 03:15:15 on dist-test-slave-nbjg"
server_key: "5555d4767699ec1752a57ec06059900f"
server_key_iv: "a9a1725aa2d7f41c227f4a6e3c9b0707"
server_key_version: "encryptionkey@0"
I20260929 03:15:15.484576 14648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:15.505920 14648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:15.508738 14648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:15.508924 14648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:15.509516 14648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:15.510366 14648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:15.510413 14648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:15.510465 14648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:15.510501 14648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:15.519025 14648 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:46343
I20260929 03:15:15.519055 14776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:46343 every 8 connection(s)
I20260929 03:15:15.519915 14648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:15:15.522570 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 14648
I20260929 03:15:15.522668 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance
I20260929 03:15:15.522935 14266 external_mini_cluster.cc:1428] Setting key 7f7ffe5c5cb3c63d788f54ea4a73ba25
Sep 29 03:15:15 dist-test-slave-nbjg krb5kdc[14273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651715, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:15.533408 14357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:15.521800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:37879 (local address 127.13.238.190:44667)
0929 03:15:15.522008 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:15.522013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:15.522932 (+ 919us) server_negotiation.cc:407] Connection header received
0929 03:15:15.524196 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:15.524199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:15.524255 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:15.524353 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:15.525680 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:15.526279 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:15.527526 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:15.527744 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:15.527825 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:15.530119 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:15.530122 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:15.530124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:15.530150 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:15.531504 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:15.531974 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:15.531976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:15.531978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:15.532057 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:15.532421 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:15.532423 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:15.532424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:15.532588 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:15.532656 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:15.533169 (+ 513us) server_negotiation.cc:299] Negotiation successful
0929 03:15:15.533273 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260929 03:15:15.534312 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:15.522276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44667 (local address 127.13.238.131:37879)
0929 03:15:15.522754 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:15.522807 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:15.524000 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:15.524564 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:15.524571 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:15.524930 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:15.525550 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:15.525568 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:15.526396 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:15.526399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:15.527406 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:15.527416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:15.527870 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:15.527873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:15.528015 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:15.528546 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:15.528562 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:15.529953 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:15.531613 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:15.531618 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:15.531633 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:15.531892 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:15.532163 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:15.532166 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:15.532167 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:15.532317 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:15.532670 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:15.532675 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:15.532941 (+ 266us) client_negotiation.cc:769] Sending connection context
0929 03:15:15.533107 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 03:15:15.533395 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":96,"threads_started":1}
I20260929 03:15:15.535583 14777 heartbeater.cc:347] Connected to a master server at 127.13.238.190:44667
I20260929 03:15:15.535848 14777 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:15.536469 14777 heartbeater.cc:511] Master 127.13.238.190:44667 requested a full tablet report, sending...
I20260929 03:15:15.537595 14318 ts_manager.cc:194] Registered new tserver with Master: fe8818986f7e467aa1074954df3f0201 (127.13.238.131:46343)
I20260929 03:15:15.538179 14318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.131@KRBTEST.COM'} at 127.13.238.131:37879
I20260929 03:15:15.549471 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:15:15 dist-test-slave-nbjg krb5kdc[14273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651715, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:15:15 dist-test-slave-nbjg krb5kdc[14273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651715, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.238.129@KRBTEST.COM
I20260929 03:15:15.574725 14784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:15.565373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54626 (local address 127.13.238.129:42909)
0929 03:15:15.565673 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:15.565687 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:15.565714 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:15:15.565871 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:15.565878 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:15.566129 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:15.566311 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:15.566920 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:15.567755 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:15.568594 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:15.568962 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:15.569057 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:15.571548 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:15.571557 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:15.571570 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:15.571593 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:15.573071 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:15.573457 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:15.573462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:15.573463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:15.573539 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:15.573752 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:15.573754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:15.573755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:15.573996 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:15.574155 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:15.574392 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 03:15:15.574530 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":97,"threads_started":1}
W20260929 03:15:15.575728 14423 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54626
Sep 29 03:15:15 dist-test-slave-nbjg krb5kdc[14273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:15:15 dist-test-slave-nbjg krb5kdc[14273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.129@KRBTEST.COM
I20260929 03:15:15.600543 14784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:15.592547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54634 (local address 127.13.238.129:42909)
0929 03:15:15.592717 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:15.592722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:15.592836 (+ 114us) server_negotiation.cc:407] Connection header received
0929 03:15:15.592977 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:15.592981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:15.593029 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:15.593105 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:15.593807 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:15.594371 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:15.595203 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:15.595437 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:15.595535 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:15.597754 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:15.597757 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:15.597759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:15.597796 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:15.598988 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:15.599389 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:15.599392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:15.599393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:15.599481 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:15.599733 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:15.599736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:15.599738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:15.599902 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:15.600017 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:15.600253 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 03:15:15.600375 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:15:15.601519 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 14380
I20260929 03:15:15.607076 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 14514
I20260929 03:15:15.611569 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 14648
I20260929 03:15:15.616043 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 14286
2026-09-29T03:15:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2075 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:15:15 dist-test-slave-nbjg krb5kdc[14791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:15:15 dist-test-slave-nbjg krb5kdc[14791](info): set up 2 sockets
Sep 29 03:15:15 dist-test-slave-nbjg krb5kdc[14791](info): commencing operation
krb5kdc: starting...
W20260929 03:15:16.561997 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:15:16 dist-test-slave-nbjg krb5kdc[14791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:15:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:15:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:15:16.679674 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:43091
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:34991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:43091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:16.788863 14804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:16.789153 14804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:16.789209 14804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:16.793303 14804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:16.793375 14804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:16.793402 14804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:16.793423 14804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:16.793443 14804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:15:16.798906 14804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:43091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:43091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:16.799989 14804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:16.801043 14804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:16.809352 14810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:16.809330 14809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:16.809342 14812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:16.809751 14804 server_base.cc:1034] running on GCE node
I20260929 03:15:16.810348 14804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:16.811436 14804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:16.812608 14804 hybrid_clock.cc:648] HybridClock initialized: now 1790651716812591 us; error 29 us; skew 500 ppm
Sep 29 03:15:16 dist-test-slave-nbjg krb5kdc[14791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651716, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:16.815444 14804 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:15:16.816361 14804 webserver.cc:460] Webserver started at http://127.13.238.190:32945/ using document root <none> and password file <none>
I20260929 03:15:16.816924 14804 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:16.817003 14804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:16.817206 14804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:16.819437 14804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "a58618748ad94490b781955a6688e6e7"
format_stamp: "Formatted at 2026-09-29 03:15:16 on dist-test-slave-nbjg"
server_key: "26c8ff7a967a9dfef6d9264dc60f2fc8"
server_key_iv: "3e0cec6b846a620040531f665048a3ca"
server_key_version: "encryptionkey@0"
I20260929 03:15:16.820011 14804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "a58618748ad94490b781955a6688e6e7"
format_stamp: "Formatted at 2026-09-29 03:15:16 on dist-test-slave-nbjg"
server_key: "26c8ff7a967a9dfef6d9264dc60f2fc8"
server_key_iv: "3e0cec6b846a620040531f665048a3ca"
server_key_version: "encryptionkey@0"
I20260929 03:15:16.823995 14804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:15:16.826624 14819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:16.827965 14804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:15:16.828090 14804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "a58618748ad94490b781955a6688e6e7"
format_stamp: "Formatted at 2026-09-29 03:15:16 on dist-test-slave-nbjg"
server_key: "26c8ff7a967a9dfef6d9264dc60f2fc8"
server_key_iv: "3e0cec6b846a620040531f665048a3ca"
server_key_version: "encryptionkey@0"
I20260929 03:15:16.828194 14804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:16.851462 14804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:16.854717 14804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:16.854912 14804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:16.861728 14804 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:43091
I20260929 03:15:16.861747 14871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:43091 every 8 connection(s)
I20260929 03:15:16.862687 14804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:15:16.865460 14872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:16.865677 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 14804
I20260929 03:15:16.865772 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:15:16.866000 14266 external_mini_cluster.cc:1428] Setting key 0ce2d550bc50b7d4dcf30c67ec2505e2
I20260929 03:15:16.871661 14872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58618748ad94490b781955a6688e6e7: Bootstrap starting.
I20260929 03:15:16.874066 14872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a58618748ad94490b781955a6688e6e7: Neither blocks nor log segments found. Creating new log.
Sep 29 03:15:16 dist-test-slave-nbjg krb5kdc[14791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:16.874925 14872 log.cc:824] T 00000000000000000000000000000000 P a58618748ad94490b781955a6688e6e7: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:16.876906 14872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a58618748ad94490b781955a6688e6e7: No bootstrap required, opened a new log
I20260929 03:15:16.879390 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:16.867313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38716 (local address 127.13.238.190:43091)
0929 03:15:16.867754 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:16.867768 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:16.867804 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:15:16.868555 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:16.868573 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:16.868908 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:16.869214 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:16.869962 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:16.871019 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:16.871843 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:16.872519 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:16.872690 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:16.875161 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:16.875166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:16.875180 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:16.875207 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:16.876731 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:16.877153 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:16.877159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:16.877164 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:16.877271 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:16.877529 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:16.877533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:16.877534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:16.877921 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:16.878092 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:16.878429 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 03:15:16.878695 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":304,"thread_start_us":130,"threads_started":1}
I20260929 03:15:16.880591 14872 raft_consensus.cc:348] T 00000000000000000000000000000000 P a58618748ad94490b781955a6688e6e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58618748ad94490b781955a6688e6e7" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43091 } }
I20260929 03:15:16.880802 14872 raft_consensus.cc:374] T 00000000000000000000000000000000 P a58618748ad94490b781955a6688e6e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:16.880856 14872 raft_consensus.cc:729] T 00000000000000000000000000000000 P a58618748ad94490b781955a6688e6e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a58618748ad94490b781955a6688e6e7, State: Initialized, Role: FOLLOWER
I20260929 03:15:16.881284 14872 consensus_queue.cc:260] T 00000000000000000000000000000000 P a58618748ad94490b781955a6688e6e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58618748ad94490b781955a6688e6e7" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43091 } }
I20260929 03:15:16.881414 14872 raft_consensus.cc:388] T 00000000000000000000000000000000 P a58618748ad94490b781955a6688e6e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:15:16.881479 14872 raft_consensus.cc:482] T 00000000000000000000000000000000 P a58618748ad94490b781955a6688e6e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:15:16.881568 14872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a58618748ad94490b781955a6688e6e7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:16.882457 14872 raft_consensus.cc:504] T 00000000000000000000000000000000 P a58618748ad94490b781955a6688e6e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58618748ad94490b781955a6688e6e7" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43091 } }
I20260929 03:15:16.882755 14872 leader_election.cc:302] T 00000000000000000000000000000000 P a58618748ad94490b781955a6688e6e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a58618748ad94490b781955a6688e6e7; no voters:
I20260929 03:15:16.883028 14872 leader_election.cc:288] T 00000000000000000000000000000000 P a58618748ad94490b781955a6688e6e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:15:16.883112 14877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a58618748ad94490b781955a6688e6e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:16.883281 14877 raft_consensus.cc:686] T 00000000000000000000000000000000 P a58618748ad94490b781955a6688e6e7 [term 1 LEADER]: Becoming Leader. State: Replica: a58618748ad94490b781955a6688e6e7, State: Running, Role: LEADER
I20260929 03:15:16.883572 14877 consensus_queue.cc:237] T 00000000000000000000000000000000 P a58618748ad94490b781955a6688e6e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58618748ad94490b781955a6688e6e7" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43091 } }
I20260929 03:15:16.884262 14872 sys_catalog.cc:564] T 00000000000000000000000000000000 P a58618748ad94490b781955a6688e6e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:16.885221 14879 sys_catalog.cc:455] T 00000000000000000000000000000000 P a58618748ad94490b781955a6688e6e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a58618748ad94490b781955a6688e6e7. Latest consensus state: current_term: 1 leader_uuid: "a58618748ad94490b781955a6688e6e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58618748ad94490b781955a6688e6e7" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43091 } } }
I20260929 03:15:16.885362 14879 sys_catalog.cc:458] T 00000000000000000000000000000000 P a58618748ad94490b781955a6688e6e7 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:16.885663 14878 sys_catalog.cc:455] T 00000000000000000000000000000000 P a58618748ad94490b781955a6688e6e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a58618748ad94490b781955a6688e6e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a58618748ad94490b781955a6688e6e7" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43091 } } }
I20260929 03:15:16.885751 14878 sys_catalog.cc:458] T 00000000000000000000000000000000 P a58618748ad94490b781955a6688e6e7 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:16.885861 14886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:15:16.888494 14886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:15:16.893814 14886 catalog_manager.cc:1347] Generated new cluster ID: c199e1409d0145c2b7095999db0f2507
I20260929 03:15:16.893882 14886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:15:16.917459 14886 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:15:16.918829 14886 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:15:16.924250 14886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a58618748ad94490b781955a6688e6e7: Generated new TSK 0
I20260929 03:15:16.925069 14886 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
I20260929 03:15:16.952781 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:0
--local_ip_for_outbound_sockets=127.13.238.129
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43091
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:34991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:17.065364 14898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:17.065631 14898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:17.065675 14898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:17.069782 14898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:17.069840 14898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:17.069904 14898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:15:17.075063 14898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43091
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:17.076018 14898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:17.076980 14898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:17.086275 14903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:17.086262 14904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:17.086282 14906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:17.086503 14898 server_base.cc:1034] running on GCE node
I20260929 03:15:17.086995 14898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:17.087592 14898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:17.088743 14898 hybrid_clock.cc:648] HybridClock initialized: now 1790651717088721 us; error 30 us; skew 500 ppm
Sep 29 03:15:17 dist-test-slave-nbjg krb5kdc[14791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651717, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:17.091432 14898 init.cc:377] Logged in from keytab as kudu/127.13.238.129@KRBTEST.COM (short username kudu)
I20260929 03:15:17.092315 14898 webserver.cc:460] Webserver started at http://127.13.238.129:41397/ using document root <none> and password file <none>
I20260929 03:15:17.092842 14898 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:17.092916 14898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:17.093097 14898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:17.095222 14898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-0/data/instance:
uuid: "b5862d5362054c78ba56299e28dba09b"
format_stamp: "Formatted at 2026-09-29 03:15:17 on dist-test-slave-nbjg"
server_key: "f2030cebefdfacfa6747ad7767a513bb"
server_key_iv: "a32508dfdb5b5e8c5a89fa9f6ed1821f"
server_key_version: "encryptionkey@0"
I20260929 03:15:17.095752 14898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance:
uuid: "b5862d5362054c78ba56299e28dba09b"
format_stamp: "Formatted at 2026-09-29 03:15:17 on dist-test-slave-nbjg"
server_key: "f2030cebefdfacfa6747ad7767a513bb"
server_key_iv: "a32508dfdb5b5e8c5a89fa9f6ed1821f"
server_key_version: "encryptionkey@0"
I20260929 03:15:17.099493 14898 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:15:17.101835 14913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:17.103025 14898 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:17.103142 14898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "b5862d5362054c78ba56299e28dba09b"
format_stamp: "Formatted at 2026-09-29 03:15:17 on dist-test-slave-nbjg"
server_key: "f2030cebefdfacfa6747ad7767a513bb"
server_key_iv: "a32508dfdb5b5e8c5a89fa9f6ed1821f"
server_key_version: "encryptionkey@0"
I20260929 03:15:17.103235 14898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:17.121423 14898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:17.124538 14898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:17.124709 14898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:17.125288 14898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:17.126163 14898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:17.126236 14898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:17.126295 14898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:17.126324 14898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:17.135277 14898 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:37479
I20260929 03:15:17.135303 15026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:37479 every 8 connection(s)
I20260929 03:15:17.136155 14898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:15:17.138882 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 14898
I20260929 03:15:17.139014 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance
I20260929 03:15:17.139284 14266 external_mini_cluster.cc:1428] Setting key d82926c1c5f586d04d6d875d4d8f3991
Sep 29 03:15:17 dist-test-slave-nbjg krb5kdc[14791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651717, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.238.130@KRBTEST.COM; defaulting to no policy
I20260929 03:15:17.150321 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:17.137900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:57825 (local address 127.13.238.190:43091)
0929 03:15:17.138040 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:17.138043 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:17.138984 (+ 941us) server_negotiation.cc:407] Connection header received
0929 03:15:17.140048 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:17.140052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:17.140096 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:17.140176 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:17.141718 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.142396 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.143544 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.143880 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.143991 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:17.146586 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:17.146590 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:17.146593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:17.146623 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:17.148223 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:17.148842 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:17.148846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:17.148847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:17.148936 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:17.149231 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:17.149233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:17.149234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:17.149414 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:17.149487 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:17.150038 (+ 551us) server_negotiation.cc:299] Negotiation successful
0929 03:15:17.150145 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":59}
I20260929 03:15:17.151278 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:17.138315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43091 (local address 127.13.238.129:57825)
0929 03:15:17.138790 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:17.138850 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:17.139825 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:17.140324 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:17.140331 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:17.140807 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:17.141481 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:17.141503 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.142518 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.142521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:17.143407 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:17.143416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.144018 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.144021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:17.144163 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:17.144771 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:17.144787 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:17.146404 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:17.148341 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:17.148345 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:17.148359 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:17.148748 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:17.149017 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:17.149019 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:17.149021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:17.149159 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:17.149512 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:17.149519 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:17.149821 (+ 302us) client_negotiation.cc:769] Sending connection context
0929 03:15:17.149992 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 03:15:17.150284 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.130@KRBTEST.COM" created.
I20260929 03:15:17.152591 15027 heartbeater.cc:347] Connected to a master server at 127.13.238.190:43091
I20260929 03:15:17.152889 15027 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:17.153393 15027 heartbeater.cc:511] Master 127.13.238.190:43091 requested a full tablet report, sending...
I20260929 03:15:17.155067 14836 ts_manager.cc:194] Registered new tserver with Master: b5862d5362054c78ba56299e28dba09b (127.13.238.129:37479)
I20260929 03:15:17.156241 14836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.129@KRBTEST.COM'} at 127.13.238.129:57825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
I20260929 03:15:17.166822 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:0
--local_ip_for_outbound_sockets=127.13.238.130
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43091
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:34991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:17.278771 15032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:17.279027 15032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:17.279083 15032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:17.283041 15032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:17.283094 15032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:17.283170 15032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:15:17.288739 15032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43091
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:17.289763 15032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:17.290804 15032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:17.300045 15040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:17.300047 15037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:17.300177 15032 server_base.cc:1034] running on GCE node
W20260929 03:15:17.300069 15038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:17.300642 15032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:17.301257 15032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:17.302409 15032 hybrid_clock.cc:648] HybridClock initialized: now 1790651717302387 us; error 30 us; skew 500 ppm
Sep 29 03:15:17 dist-test-slave-nbjg krb5kdc[14791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651717, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:17.304932 15032 init.cc:377] Logged in from keytab as kudu/127.13.238.130@KRBTEST.COM (short username kudu)
I20260929 03:15:17.305743 15032 webserver.cc:460] Webserver started at http://127.13.238.130:45621/ using document root <none> and password file <none>
I20260929 03:15:17.306330 15032 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:17.306392 15032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:17.306588 15032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:17.308636 15032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-1/data/instance:
uuid: "77f6ae2777d944139bc5785e1fd8a7e2"
format_stamp: "Formatted at 2026-09-29 03:15:17 on dist-test-slave-nbjg"
server_key: "be9a1b7e21c6d195e496a31a6d0ad305"
server_key_iv: "7836a60b4ab4f7cf418bf2a19412c9d7"
server_key_version: "encryptionkey@0"
I20260929 03:15:17.309165 15032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance:
uuid: "77f6ae2777d944139bc5785e1fd8a7e2"
format_stamp: "Formatted at 2026-09-29 03:15:17 on dist-test-slave-nbjg"
server_key: "be9a1b7e21c6d195e496a31a6d0ad305"
server_key_iv: "7836a60b4ab4f7cf418bf2a19412c9d7"
server_key_version: "encryptionkey@0"
I20260929 03:15:17.312830 15032 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 03:15:17.315200 15047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:17.316429 15032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:17.316527 15032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "77f6ae2777d944139bc5785e1fd8a7e2"
format_stamp: "Formatted at 2026-09-29 03:15:17 on dist-test-slave-nbjg"
server_key: "be9a1b7e21c6d195e496a31a6d0ad305"
server_key_iv: "7836a60b4ab4f7cf418bf2a19412c9d7"
server_key_version: "encryptionkey@0"
I20260929 03:15:17.316627 15032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:17.339500 15032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:17.342456 15032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:17.342638 15032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:17.343230 15032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:17.344033 15032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:17.344075 15032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:17.344118 15032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:17.344146 15032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:17.352417 15032 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:39199
I20260929 03:15:17.352454 15160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:39199 every 8 connection(s)
I20260929 03:15:17.353297 15032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:15:17 dist-test-slave-nbjg krb5kdc[14791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651717, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:17.362928 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 15032
I20260929 03:15:17.363042 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance
I20260929 03:15:17.363322 14266 external_mini_cluster.cc:1428] Setting key 94b031540becfbbfcebc89304720f92f
I20260929 03:15:17.366258 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:17.355107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:36847 (local address 127.13.238.190:43091)
0929 03:15:17.355279 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:17.355284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:17.356128 (+ 844us) server_negotiation.cc:407] Connection header received
0929 03:15:17.356988 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:17.356992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:17.357045 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:17.357132 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:17.358297 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.358822 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.359983 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.360207 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.360287 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:17.362733 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:17.362736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:17.362738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:17.362763 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:17.364240 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:17.364775 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:17.364779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:17.364781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:17.364859 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:17.365177 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:17.365180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:17.365182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:17.365363 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:17.365445 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:17.365948 (+ 503us) server_negotiation.cc:299] Negotiation successful
0929 03:15:17.366072 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 03:15:17.367090 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:17.355516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43091 (local address 127.13.238.130:36847)
0929 03:15:17.355968 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:17.356011 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:17.356762 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:17.357297 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:17.357304 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:17.357647 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:17.358144 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:17.358158 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.358932 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.358935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:17.359849 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:17.359859 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.360363 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.360367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:17.360513 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:17.361068 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:17.361089 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:17.362542 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:17.364346 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:17.364352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:17.364369 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:17.364678 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:17.364942 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:17.364945 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:17.364947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:17.365087 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:17.365463 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:17.365471 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:17.365717 (+ 246us) client_negotiation.cc:769] Sending connection context
0929 03:15:17.365886 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 03:15:17.366259 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":105,"threads_started":1}
I20260929 03:15:17.368206 15161 heartbeater.cc:347] Connected to a master server at 127.13.238.190:43091
I20260929 03:15:17.368436 15161 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:17.368901 15161 heartbeater.cc:511] Master 127.13.238.190:43091 requested a full tablet report, sending...
I20260929 03:15:17.369892 14836 ts_manager.cc:194] Registered new tserver with Master: 77f6ae2777d944139bc5785e1fd8a7e2 (127.13.238.130:39199)
I20260929 03:15:17.370522 14836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.130@KRBTEST.COM'} at 127.13.238.130:36847
WARNING: no policy specified for kudu/127.13.238.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
I20260929 03:15:17.389716 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:0
--local_ip_for_outbound_sockets=127.13.238.131
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43091
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:34991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:17.500293 15166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:17.500540 15166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:17.500581 15166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:17.504560 15166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:17.504616 15166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:17.504674 15166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:15:17.510000 15166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43091
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:17.511024 15166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:17.512004 15166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:17.521466 15174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:17.521596 15166 server_base.cc:1034] running on GCE node
W20260929 03:15:17.521522 15172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:17.521607 15171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:17.522321 15166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:17.522943 15166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:17.524109 15166 hybrid_clock.cc:648] HybridClock initialized: now 1790651717524087 us; error 36 us; skew 500 ppm
Sep 29 03:15:17 dist-test-slave-nbjg krb5kdc[14791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651717, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:17.526749 15166 init.cc:377] Logged in from keytab as kudu/127.13.238.131@KRBTEST.COM (short username kudu)
I20260929 03:15:17.527595 15166 webserver.cc:460] Webserver started at http://127.13.238.131:38575/ using document root <none> and password file <none>
I20260929 03:15:17.528126 15166 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:17.528191 15166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:17.528378 15166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:17.530435 15166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-2/data/instance:
uuid: "1dcda0a53ee644a9a6de3ec2f38c302c"
format_stamp: "Formatted at 2026-09-29 03:15:17 on dist-test-slave-nbjg"
server_key: "148648c5e9dfbc264a0e10bf6dd06887"
server_key_iv: "7cdac29ec2d23cb66b66c513af1adb5a"
server_key_version: "encryptionkey@0"
I20260929 03:15:17.530966 15166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance:
uuid: "1dcda0a53ee644a9a6de3ec2f38c302c"
format_stamp: "Formatted at 2026-09-29 03:15:17 on dist-test-slave-nbjg"
server_key: "148648c5e9dfbc264a0e10bf6dd06887"
server_key_iv: "7cdac29ec2d23cb66b66c513af1adb5a"
server_key_version: "encryptionkey@0"
I20260929 03:15:17.534687 15166 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:15:17.537073 15181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:17.538331 15166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:17.538434 15166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "1dcda0a53ee644a9a6de3ec2f38c302c"
format_stamp: "Formatted at 2026-09-29 03:15:17 on dist-test-slave-nbjg"
server_key: "148648c5e9dfbc264a0e10bf6dd06887"
server_key_iv: "7cdac29ec2d23cb66b66c513af1adb5a"
server_key_version: "encryptionkey@0"
I20260929 03:15:17.538538 15166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:17.554648 15166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:17.557988 15166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:17.558182 15166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:17.558826 15166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:17.559697 15166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:17.559746 15166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:17.559800 15166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:17.559829 15166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:17.568324 15166 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:37203
I20260929 03:15:17.568410 15294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:37203 every 8 connection(s)
I20260929 03:15:17.569249 15166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:15:17.575312 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 15166
I20260929 03:15:17.575423 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance
I20260929 03:15:17.575661 14266 external_mini_cluster.cc:1428] Setting key 3eac62efc3f5960c60243a9547fa42ad
Sep 29 03:15:17 dist-test-slave-nbjg krb5kdc[14791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651717, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:17.581928 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:17.571216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:34269 (local address 127.13.238.190:43091)
0929 03:15:17.571365 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:17.571368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:17.572206 (+ 838us) server_negotiation.cc:407] Connection header received
0929 03:15:17.573100 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:17.573103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:17.573158 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:17.573232 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:17.574460 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.575000 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.576207 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.576516 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.576616 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:17.579165 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:17.579169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:17.579171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:17.579195 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:17.580285 (+ 1090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:17.580731 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:17.580733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:17.580734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:17.580800 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:17.581052 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:17.581054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:17.581055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:17.581212 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:17.581281 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:17.581703 (+ 422us) server_negotiation.cc:299] Negotiation successful
0929 03:15:17.581805 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 03:15:17.582748 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:17.571612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43091 (local address 127.13.238.131:34269)
0929 03:15:17.572055 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:17.572098 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:17.572878 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:17.573429 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:17.573436 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:17.573804 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:17.574311 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:17.574326 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.575126 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.575128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:17.576088 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:17.576100 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.576669 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.576673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:17.576855 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:17.577448 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:17.577465 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:17.579020 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:17.580402 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:17.580407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:17.580419 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:17.580658 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:17.580861 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:17.580863 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:17.580865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:17.580990 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:17.581281 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:17.581285 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:17.581505 (+ 220us) client_negotiation.cc:769] Sending connection context
0929 03:15:17.581654 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 03:15:17.581901 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":106,"threads_started":1}
I20260929 03:15:17.583758 15295 heartbeater.cc:347] Connected to a master server at 127.13.238.190:43091
I20260929 03:15:17.583962 15295 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:17.584394 15295 heartbeater.cc:511] Master 127.13.238.190:43091 requested a full tablet report, sending...
I20260929 03:15:17.585332 14836 ts_manager.cc:194] Registered new tserver with Master: 1dcda0a53ee644a9a6de3ec2f38c302c (127.13.238.131:37203)
I20260929 03:15:17.585927 14836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.131@KRBTEST.COM'} at 127.13.238.131:34269
I20260929 03:15:17.589214 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:15:17.598793 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:17.592464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38730 (local address 127.13.238.190:43091)
0929 03:15:17.592606 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:17.592610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:17.592707 (+ 97us) server_negotiation.cc:407] Connection header received
0929 03:15:17.592797 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:17.592799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:17.592837 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:17.592898 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:17.593485 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.593970 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.594764 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.594992 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.595070 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:17.596182 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:17.596185 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:17.596187 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:17.596207 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:17.597437 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:17.597775 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:17.597778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:17.597780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:17.597845 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:17.598027 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:17.598029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:17.598030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:17.598269 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:17.598364 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:17.598545 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 03:15:17.598634 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 03:15:17.603387 14836 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:15:17.606336 14836 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:15:17.620951 15309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:17.615443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:37203 (local address 127.0.0.1:44502)
0929 03:15:17.616188 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:17.616200 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:17.616264 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:17.616964 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:17.616967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:17.616984 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:17.617087 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:17.617094 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.618115 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.618117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:17.619223 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:17.619234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.620343 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.620347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:17.620476 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:17.620521 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 03:15:17.620654 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 03:15:17.620794 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":83,"threads_started":1}
I20260929 03:15:17.621107 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:17.614760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:39199 (local address 127.0.0.1:52802)
0929 03:15:17.615095 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:17.615151 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:17.615282 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:17.616194 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:17.616202 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:17.616226 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:17.616373 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:17.616378 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.617953 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.617956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:17.619460 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:17.619472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.620636 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.620640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:17.620731 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:17.620754 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:15:17.620821 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 03:15:17.620900 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":3,"thread_start_us":67,"threads_started":1}
I20260929 03:15:17.621265 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:17.614859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52802 (local address 127.13.238.130:39199)
0929 03:15:17.615687 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:17.615699 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:17.615724 (+ 25us) server_negotiation.cc:407] Connection header received
0929 03:15:17.615802 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:17.615807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:17.616021 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:17.616497 (+ 476us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:17.616645 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.617816 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.619601 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.620479 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.620640 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:17.620763 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:17.620859 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 03:15:17.620954 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":158,"threads_started":1}
I20260929 03:15:17.621049 15310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:17.615511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38724 (local address 127.13.238.129:37479)
0929 03:15:17.615854 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:17.615870 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:17.615892 (+ 22us) server_negotiation.cc:407] Connection header received
0929 03:15:17.615990 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:17.615995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:17.616196 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:17.616352 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:17.616611 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.617825 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.619628 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.620188 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.620272 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:17.620414 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:17.620797 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 03:15:17.620908 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":60,"threads_started":1}
I20260929 03:15:17.621630 15311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:17.615825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44502 (local address 127.13.238.131:37203)
0929 03:15:17.616639 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:17.616647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:17.616664 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:15:17.616720 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:17.616724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:17.616848 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:17.616972 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:17.617204 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.617984 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.619361 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.620201 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.620330 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:17.620464 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:17.621365 (+ 901us) server_negotiation.cc:299] Negotiation successful
0929 03:15:17.621489 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":111,"threads_started":1}
I20260929 03:15:17.621662 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:17.615101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:37479 (local address 127.0.0.1:38724)
0929 03:15:17.615496 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:17.615508 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:17.615589 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:17.616337 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:17.616340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:17.616359 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:17.616484 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:17.616491 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.618280 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.618283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:17.619478 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:17.619484 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.620511 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.620514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:17.620604 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:17.620623 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 03:15:17.621478 (+ 855us) client_negotiation.cc:240] Negotiation successful
0929 03:15:17.621553 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":52,"thread_start_us":94,"threads_started":1}
I20260929 03:15:17.624115 14961 tablet_service.cc:1467] Processing CreateTablet for tablet 32cbf0aec3d64289860fe8bc2918e7be (DEFAULT_TABLE table=test-table [id=a1d6d58f7cdb4985ac5a040c93196e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:17.624114 15095 tablet_service.cc:1467] Processing CreateTablet for tablet 32cbf0aec3d64289860fe8bc2918e7be (DEFAULT_TABLE table=test-table [id=a1d6d58f7cdb4985ac5a040c93196e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:17.625038 14961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32cbf0aec3d64289860fe8bc2918e7be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:17.625039 15095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32cbf0aec3d64289860fe8bc2918e7be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:17.631445 15312 tablet_bootstrap.cc:492] T 32cbf0aec3d64289860fe8bc2918e7be P 77f6ae2777d944139bc5785e1fd8a7e2: Bootstrap starting.
I20260929 03:15:17.632413 15229 tablet_service.cc:1467] Processing CreateTablet for tablet 32cbf0aec3d64289860fe8bc2918e7be (DEFAULT_TABLE table=test-table [id=a1d6d58f7cdb4985ac5a040c93196e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:17.633563 15229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32cbf0aec3d64289860fe8bc2918e7be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:17.633968 15312 tablet_bootstrap.cc:654] T 32cbf0aec3d64289860fe8bc2918e7be P 77f6ae2777d944139bc5785e1fd8a7e2: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:17.633968 15313 tablet_bootstrap.cc:492] T 32cbf0aec3d64289860fe8bc2918e7be P b5862d5362054c78ba56299e28dba09b: Bootstrap starting.
I20260929 03:15:17.634895 15312 log.cc:824] T 32cbf0aec3d64289860fe8bc2918e7be P 77f6ae2777d944139bc5785e1fd8a7e2: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:17.636454 15313 tablet_bootstrap.cc:654] T 32cbf0aec3d64289860fe8bc2918e7be P b5862d5362054c78ba56299e28dba09b: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:17.637245 15312 tablet_bootstrap.cc:492] T 32cbf0aec3d64289860fe8bc2918e7be P 77f6ae2777d944139bc5785e1fd8a7e2: No bootstrap required, opened a new log
I20260929 03:15:17.637436 15312 ts_tablet_manager.cc:1397] T 32cbf0aec3d64289860fe8bc2918e7be P 77f6ae2777d944139bc5785e1fd8a7e2: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260929 03:15:17.637661 15313 log.cc:824] T 32cbf0aec3d64289860fe8bc2918e7be P b5862d5362054c78ba56299e28dba09b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:17.639771 15313 tablet_bootstrap.cc:492] T 32cbf0aec3d64289860fe8bc2918e7be P b5862d5362054c78ba56299e28dba09b: No bootstrap required, opened a new log
I20260929 03:15:17.639957 15313 ts_tablet_manager.cc:1397] T 32cbf0aec3d64289860fe8bc2918e7be P b5862d5362054c78ba56299e28dba09b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 03:15:17.640233 15316 tablet_bootstrap.cc:492] T 32cbf0aec3d64289860fe8bc2918e7be P 1dcda0a53ee644a9a6de3ec2f38c302c: Bootstrap starting.
I20260929 03:15:17.641209 15312 raft_consensus.cc:348] T 32cbf0aec3d64289860fe8bc2918e7be P 77f6ae2777d944139bc5785e1fd8a7e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77f6ae2777d944139bc5785e1fd8a7e2" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 39199 } } peers { permanent_uuid: "1dcda0a53ee644a9a6de3ec2f38c302c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 37203 } } peers { permanent_uuid: "b5862d5362054c78ba56299e28dba09b" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 37479 } }
I20260929 03:15:17.641492 15312 raft_consensus.cc:374] T 32cbf0aec3d64289860fe8bc2918e7be P 77f6ae2777d944139bc5785e1fd8a7e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:17.641551 15312 raft_consensus.cc:729] T 32cbf0aec3d64289860fe8bc2918e7be P 77f6ae2777d944139bc5785e1fd8a7e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77f6ae2777d944139bc5785e1fd8a7e2, State: Initialized, Role: FOLLOWER
I20260929 03:15:17.642017 15312 consensus_queue.cc:260] T 32cbf0aec3d64289860fe8bc2918e7be P 77f6ae2777d944139bc5785e1fd8a7e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77f6ae2777d944139bc5785e1fd8a7e2" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 39199 } } peers { permanent_uuid: "1dcda0a53ee644a9a6de3ec2f38c302c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 37203 } } peers { permanent_uuid: "b5862d5362054c78ba56299e28dba09b" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 37479 } }
I20260929 03:15:17.642793 15161 heartbeater.cc:503] Master 127.13.238.190:43091 was elected leader, sending a full tablet report...
I20260929 03:15:17.642851 15316 tablet_bootstrap.cc:654] T 32cbf0aec3d64289860fe8bc2918e7be P 1dcda0a53ee644a9a6de3ec2f38c302c: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:17.643093 15312 ts_tablet_manager.cc:1428] T 32cbf0aec3d64289860fe8bc2918e7be P 77f6ae2777d944139bc5785e1fd8a7e2: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 03:15:17.643585 15316 log.cc:824] T 32cbf0aec3d64289860fe8bc2918e7be P 1dcda0a53ee644a9a6de3ec2f38c302c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:17.644819 15313 raft_consensus.cc:348] T 32cbf0aec3d64289860fe8bc2918e7be P b5862d5362054c78ba56299e28dba09b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77f6ae2777d944139bc5785e1fd8a7e2" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 39199 } } peers { permanent_uuid: "1dcda0a53ee644a9a6de3ec2f38c302c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 37203 } } peers { permanent_uuid: "b5862d5362054c78ba56299e28dba09b" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 37479 } }
I20260929 03:15:17.645073 15313 raft_consensus.cc:374] T 32cbf0aec3d64289860fe8bc2918e7be P b5862d5362054c78ba56299e28dba09b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:17.645167 15313 raft_consensus.cc:729] T 32cbf0aec3d64289860fe8bc2918e7be P b5862d5362054c78ba56299e28dba09b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5862d5362054c78ba56299e28dba09b, State: Initialized, Role: FOLLOWER
I20260929 03:15:17.645583 15313 consensus_queue.cc:260] T 32cbf0aec3d64289860fe8bc2918e7be P b5862d5362054c78ba56299e28dba09b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77f6ae2777d944139bc5785e1fd8a7e2" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 39199 } } peers { permanent_uuid: "1dcda0a53ee644a9a6de3ec2f38c302c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 37203 } } peers { permanent_uuid: "b5862d5362054c78ba56299e28dba09b" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 37479 } }
I20260929 03:15:17.646225 15316 tablet_bootstrap.cc:492] T 32cbf0aec3d64289860fe8bc2918e7be P 1dcda0a53ee644a9a6de3ec2f38c302c: No bootstrap required, opened a new log
I20260929 03:15:17.646422 15316 ts_tablet_manager.cc:1397] T 32cbf0aec3d64289860fe8bc2918e7be P 1dcda0a53ee644a9a6de3ec2f38c302c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.003s
I20260929 03:15:17.646626 15027 heartbeater.cc:503] Master 127.13.238.190:43091 was elected leader, sending a full tablet report...
I20260929 03:15:17.646670 15313 ts_tablet_manager.cc:1428] T 32cbf0aec3d64289860fe8bc2918e7be P b5862d5362054c78ba56299e28dba09b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 03:15:17.649938 15316 raft_consensus.cc:348] T 32cbf0aec3d64289860fe8bc2918e7be P 1dcda0a53ee644a9a6de3ec2f38c302c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77f6ae2777d944139bc5785e1fd8a7e2" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 39199 } } peers { permanent_uuid: "1dcda0a53ee644a9a6de3ec2f38c302c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 37203 } } peers { permanent_uuid: "b5862d5362054c78ba56299e28dba09b" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 37479 } }
I20260929 03:15:17.650213 15316 raft_consensus.cc:374] T 32cbf0aec3d64289860fe8bc2918e7be P 1dcda0a53ee644a9a6de3ec2f38c302c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:17.650271 15316 raft_consensus.cc:729] T 32cbf0aec3d64289860fe8bc2918e7be P 1dcda0a53ee644a9a6de3ec2f38c302c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dcda0a53ee644a9a6de3ec2f38c302c, State: Initialized, Role: FOLLOWER
I20260929 03:15:17.650647 15316 consensus_queue.cc:260] T 32cbf0aec3d64289860fe8bc2918e7be P 1dcda0a53ee644a9a6de3ec2f38c302c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77f6ae2777d944139bc5785e1fd8a7e2" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 39199 } } peers { permanent_uuid: "1dcda0a53ee644a9a6de3ec2f38c302c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 37203 } } peers { permanent_uuid: "b5862d5362054c78ba56299e28dba09b" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 37479 } }
I20260929 03:15:17.651219 15295 heartbeater.cc:503] Master 127.13.238.190:43091 was elected leader, sending a full tablet report...
I20260929 03:15:17.651546 15316 ts_tablet_manager.cc:1428] T 32cbf0aec3d64289860fe8bc2918e7be P 1dcda0a53ee644a9a6de3ec2f38c302c: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
W20260929 03:15:17.820298 15296 tablet.cc:2369] T 32cbf0aec3d64289860fe8bc2918e7be P 1dcda0a53ee644a9a6de3ec2f38c302c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:15:17.854377 15162 tablet.cc:2369] T 32cbf0aec3d64289860fe8bc2918e7be P 77f6ae2777d944139bc5785e1fd8a7e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:15:17.887391 15028 tablet.cc:2369] T 32cbf0aec3d64289860fe8bc2918e7be P b5862d5362054c78ba56299e28dba09b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:15:17.914795 15317 raft_consensus.cc:482] T 32cbf0aec3d64289860fe8bc2918e7be P 77f6ae2777d944139bc5785e1fd8a7e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:15:17.915012 15317 raft_consensus.cc:504] T 32cbf0aec3d64289860fe8bc2918e7be P 77f6ae2777d944139bc5785e1fd8a7e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77f6ae2777d944139bc5785e1fd8a7e2" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 39199 } } peers { permanent_uuid: "1dcda0a53ee644a9a6de3ec2f38c302c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 37203 } } peers { permanent_uuid: "b5862d5362054c78ba56299e28dba09b" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 37479 } }
I20260929 03:15:17.916203 15317 leader_election.cc:288] T 32cbf0aec3d64289860fe8bc2918e7be P 77f6ae2777d944139bc5785e1fd8a7e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dcda0a53ee644a9a6de3ec2f38c302c (127.13.238.131:37203), b5862d5362054c78ba56299e28dba09b (127.13.238.129:37479)
I20260929 03:15:17.920082 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:17.916465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:37203 (local address 127.13.238.130:43223)
0929 03:15:17.916830 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:17.916843 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:17.916952 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:17.917179 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:17.917182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:17.917197 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:17.917296 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:17.917301 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.918055 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.918058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:17.918985 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:17.918995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.919627 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.919630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:17.919804 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:17.919826 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 03:15:17.919884 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 03:15:17.919934 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":141,"thread_start_us":78,"threads_started":1}
I20260929 03:15:17.920107 15311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:17.916547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:43223 (local address 127.13.238.131:37203)
0929 03:15:17.916675 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:17.916680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:17.916943 (+ 263us) server_negotiation.cc:407] Connection header received
0929 03:15:17.917031 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:17.917033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:17.917077 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:17.917149 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:17.917411 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.917914 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.919116 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.919510 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.919623 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:17.919673 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:17.919913 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 03:15:17.919982 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:15:17.920107 15310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:17.916567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:34645 (local address 127.13.238.129:37479)
0929 03:15:17.916793 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:17.916798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:17.916814 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:15:17.916959 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:17.916962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:17.917014 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:17.917080 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:17.917373 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.917917 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.919070 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.919476 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.919567 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:17.919619 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:17.919906 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 03:15:17.919983 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260929 03:15:17.920084 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:17.916463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:37479 (local address 127.13.238.130:34645)
0929 03:15:17.916742 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:17.916757 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:17.916845 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:17.917112 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:17.917115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:17.917131 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:17.917262 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:17.917268 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.918040 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.918044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:17.918954 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:17.918964 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.919587 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.919591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:17.919682 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:17.919742 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 03:15:17.919815 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 03:15:17.919890 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":2,"thread_start_us":68,"threads_started":1}
I20260929 03:15:17.920964 14981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32cbf0aec3d64289860fe8bc2918e7be" candidate_uuid: "77f6ae2777d944139bc5785e1fd8a7e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5862d5362054c78ba56299e28dba09b" is_pre_election: true
I20260929 03:15:17.920972 15249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32cbf0aec3d64289860fe8bc2918e7be" candidate_uuid: "77f6ae2777d944139bc5785e1fd8a7e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dcda0a53ee644a9a6de3ec2f38c302c" is_pre_election: true
I20260929 03:15:17.921258 15249 raft_consensus.cc:2454] T 32cbf0aec3d64289860fe8bc2918e7be P 1dcda0a53ee644a9a6de3ec2f38c302c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77f6ae2777d944139bc5785e1fd8a7e2 in term 0.
I20260929 03:15:17.921263 14981 raft_consensus.cc:2454] T 32cbf0aec3d64289860fe8bc2918e7be P b5862d5362054c78ba56299e28dba09b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77f6ae2777d944139bc5785e1fd8a7e2 in term 0.
I20260929 03:15:17.921728 15050 leader_election.cc:302] T 32cbf0aec3d64289860fe8bc2918e7be P 77f6ae2777d944139bc5785e1fd8a7e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77f6ae2777d944139bc5785e1fd8a7e2, b5862d5362054c78ba56299e28dba09b; no voters:
I20260929 03:15:17.922010 15317 raft_consensus.cc:2781] T 32cbf0aec3d64289860fe8bc2918e7be P 77f6ae2777d944139bc5785e1fd8a7e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:15:17.922096 15317 raft_consensus.cc:482] T 32cbf0aec3d64289860fe8bc2918e7be P 77f6ae2777d944139bc5785e1fd8a7e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:15:17.922137 15317 raft_consensus.cc:3037] T 32cbf0aec3d64289860fe8bc2918e7be P 77f6ae2777d944139bc5785e1fd8a7e2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:17.923170 15317 raft_consensus.cc:504] T 32cbf0aec3d64289860fe8bc2918e7be P 77f6ae2777d944139bc5785e1fd8a7e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77f6ae2777d944139bc5785e1fd8a7e2" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 39199 } } peers { permanent_uuid: "1dcda0a53ee644a9a6de3ec2f38c302c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 37203 } } peers { permanent_uuid: "b5862d5362054c78ba56299e28dba09b" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 37479 } }
I20260929 03:15:17.923492 15317 leader_election.cc:288] T 32cbf0aec3d64289860fe8bc2918e7be P 77f6ae2777d944139bc5785e1fd8a7e2 [CANDIDATE]: Term 1 election: Requested vote from peers 1dcda0a53ee644a9a6de3ec2f38c302c (127.13.238.131:37203), b5862d5362054c78ba56299e28dba09b (127.13.238.129:37479)
I20260929 03:15:17.923882 15249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32cbf0aec3d64289860fe8bc2918e7be" candidate_uuid: "77f6ae2777d944139bc5785e1fd8a7e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dcda0a53ee644a9a6de3ec2f38c302c"
I20260929 03:15:17.923987 15249 raft_consensus.cc:3037] T 32cbf0aec3d64289860fe8bc2918e7be P 1dcda0a53ee644a9a6de3ec2f38c302c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:17.924963 15249 raft_consensus.cc:2454] T 32cbf0aec3d64289860fe8bc2918e7be P 1dcda0a53ee644a9a6de3ec2f38c302c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77f6ae2777d944139bc5785e1fd8a7e2 in term 1.
I20260929 03:15:17.925226 14981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32cbf0aec3d64289860fe8bc2918e7be" candidate_uuid: "77f6ae2777d944139bc5785e1fd8a7e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5862d5362054c78ba56299e28dba09b"
I20260929 03:15:17.925268 15051 leader_election.cc:302] T 32cbf0aec3d64289860fe8bc2918e7be P 77f6ae2777d944139bc5785e1fd8a7e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dcda0a53ee644a9a6de3ec2f38c302c, 77f6ae2777d944139bc5785e1fd8a7e2; no voters:
I20260929 03:15:17.925355 14981 raft_consensus.cc:3037] T 32cbf0aec3d64289860fe8bc2918e7be P b5862d5362054c78ba56299e28dba09b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:17.925441 15317 raft_consensus.cc:2781] T 32cbf0aec3d64289860fe8bc2918e7be P 77f6ae2777d944139bc5785e1fd8a7e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:17.925679 15317 raft_consensus.cc:686] T 32cbf0aec3d64289860fe8bc2918e7be P 77f6ae2777d944139bc5785e1fd8a7e2 [term 1 LEADER]: Becoming Leader. State: Replica: 77f6ae2777d944139bc5785e1fd8a7e2, State: Running, Role: LEADER
I20260929 03:15:17.926023 15317 consensus_queue.cc:237] T 32cbf0aec3d64289860fe8bc2918e7be P 77f6ae2777d944139bc5785e1fd8a7e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77f6ae2777d944139bc5785e1fd8a7e2" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 39199 } } peers { permanent_uuid: "1dcda0a53ee644a9a6de3ec2f38c302c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 37203 } } peers { permanent_uuid: "b5862d5362054c78ba56299e28dba09b" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 37479 } }
I20260929 03:15:17.926484 14981 raft_consensus.cc:2454] T 32cbf0aec3d64289860fe8bc2918e7be P b5862d5362054c78ba56299e28dba09b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77f6ae2777d944139bc5785e1fd8a7e2 in term 1.
I20260929 03:15:17.929363 14835 catalog_manager.cc:5495] T 32cbf0aec3d64289860fe8bc2918e7be P 77f6ae2777d944139bc5785e1fd8a7e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77f6ae2777d944139bc5785e1fd8a7e2 (127.13.238.130). New cstate: current_term: 1 leader_uuid: "77f6ae2777d944139bc5785e1fd8a7e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77f6ae2777d944139bc5785e1fd8a7e2" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 39199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dcda0a53ee644a9a6de3ec2f38c302c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 37203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5862d5362054c78ba56299e28dba09b" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 37479 } health_report { overall_health: UNKNOWN } } }
Sep 29 03:15:17 dist-test-slave-nbjg krb5kdc[14791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.129@KRBTEST.COM
I20260929 03:15:17.987250 15310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:17.978546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38726 (local address 127.13.238.129:37479)
0929 03:15:17.978699 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:17.978703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:17.978777 (+ 74us) server_negotiation.cc:407] Connection header received
0929 03:15:17.978924 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:17.978926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:17.978969 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:17.979037 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:17.979640 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.980131 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.981039 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:17.981256 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:17.981348 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:17.983768 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:17.983782 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:17.983800 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:17.983823 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:17.985517 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:17.985988 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:17.985995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:17.985996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:17.986087 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:17.986393 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:17.986396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:17.986398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:17.986669 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:17.986834 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:17.987024 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 03:15:17.987126 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 03:15:17 dist-test-slave-nbjg krb5kdc[14791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651717, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:15:18 dist-test-slave-nbjg krb5kdc[14791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651717, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.238.129@KRBTEST.COM
I20260929 03:15:18.013299 15310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:18.005428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38736 (local address 127.13.238.129:37479)
0929 03:15:18.005578 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:18.005582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:18.005687 (+ 105us) server_negotiation.cc:407] Connection header received
0929 03:15:18.005851 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:18.005854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:18.005897 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:18.005967 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:18.006676 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:18.007234 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:18.008169 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:18.008388 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:18.008467 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:18.010684 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:18.010687 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:18.010689 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:18.010715 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:18.011918 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:18.012289 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:18.012291 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:18.012292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:18.012362 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:18.012578 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:18.012580 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:18.012581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:18.012734 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:18.012834 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:18.013068 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 03:15:18.013165 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 03:15:18 dist-test-slave-nbjg krb5kdc[14791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:15:18 dist-test-slave-nbjg krb5kdc[14791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.129@KRBTEST.COM
I20260929 03:15:18.037194 15310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:18.029155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38746 (local address 127.13.238.129:37479)
0929 03:15:18.029305 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:18.029309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:18.029447 (+ 138us) server_negotiation.cc:407] Connection header received
0929 03:15:18.029540 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:18.029542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:18.029594 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:18.029670 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:18.030451 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:18.030988 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:18.031834 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:18.032084 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:18.032160 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:18.034464 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:18.034467 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:18.034469 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:18.034494 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:18.035639 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:18.036040 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:18.036043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:18.036045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:18.036121 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:18.036382 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:18.036386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:18.036388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:18.036566 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:18.036682 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:18.036921 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 03:15:18.037037 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 03:15:18.038655 14925 tablet_service.cc:2908] Rejecting scan request for tablet 32cbf0aec3d64289860fe8bc2918e7be: Uninitialized: clean time has not yet been initialized
I20260929 03:15:18.040762 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 14898
I20260929 03:15:18.046151 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 15032
I20260929 03:15:18.050926 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 15166
I20260929 03:15:18.055411 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 14804
2026-09-29T03:15:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2439 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:15:18 dist-test-slave-nbjg krb5kdc[15337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:15:18 dist-test-slave-nbjg krb5kdc[15337](info): set up 2 sockets
Sep 29 03:15:18 dist-test-slave-nbjg krb5kdc[15337](info): commencing operation
krb5kdc: starting...
W20260929 03:15:19.005972 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:15:19 dist-test-slave-nbjg krb5kdc[15337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:15:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:15:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:15:19.125000 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:37979
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:45977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:37979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:19.236696 15350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:19.236974 15350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:19.237028 15350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:19.241091 15350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:19.241143 15350 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 03:15:19.241168 15350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:19.241190 15350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:19.241219 15350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:19.241250 15350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:15:19.246487 15350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:45977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:37979
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:37979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:19.247510 15350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:19.248507 15350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:19.256644 15356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:19.256667 15358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:19.256681 15355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:19.256837 15350 server_base.cc:1034] running on GCE node
I20260929 03:15:19.257452 15350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:19.258569 15350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:19.259735 15350 hybrid_clock.cc:648] HybridClock initialized: now 1790651719259708 us; error 41 us; skew 500 ppm
Sep 29 03:15:19 dist-test-slave-nbjg krb5kdc[15337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651719, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:19.266995 15350 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:15:19.267936 15350 webserver.cc:460] Webserver started at http://127.13.238.190:42313/ using document root <none> and password file <none>
I20260929 03:15:19.268525 15350 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:19.268587 15350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:19.268784 15350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:19.270944 15350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "1324724fd0e94195be72e9cedb9c27ef"
format_stamp: "Formatted at 2026-09-29 03:15:19 on dist-test-slave-nbjg"
server_key: "a69def553905a4cfe9445c3e3ab3ea6e"
server_key_iv: "1ac418999bf5811139b869af687659e7"
server_key_version: "encryptionkey@0"
I20260929 03:15:19.271504 15350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "1324724fd0e94195be72e9cedb9c27ef"
format_stamp: "Formatted at 2026-09-29 03:15:19 on dist-test-slave-nbjg"
server_key: "a69def553905a4cfe9445c3e3ab3ea6e"
server_key_iv: "1ac418999bf5811139b869af687659e7"
server_key_version: "encryptionkey@0"
I20260929 03:15:19.275265 15350 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:15:19.277647 15365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:19.278900 15350 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:19.279013 15350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "1324724fd0e94195be72e9cedb9c27ef"
format_stamp: "Formatted at 2026-09-29 03:15:19 on dist-test-slave-nbjg"
server_key: "a69def553905a4cfe9445c3e3ab3ea6e"
server_key_iv: "1ac418999bf5811139b869af687659e7"
server_key_version: "encryptionkey@0"
I20260929 03:15:19.279112 15350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:19.295459 15350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:19.298328 15350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:19.298506 15350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:19.306658 15350 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:37979
I20260929 03:15:19.306665 15427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:37979 every 8 connection(s)
I20260929 03:15:19.307647 15350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:15:19.310511 15428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:19.310997 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 15350
I20260929 03:15:19.311106 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:15:19.311404 14266 external_mini_cluster.cc:1428] Setting key 8cb7c57f132f8ee5c36e76141099c044
I20260929 03:15:19.317407 15428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1324724fd0e94195be72e9cedb9c27ef: Bootstrap starting.
I20260929 03:15:19.319844 15428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1324724fd0e94195be72e9cedb9c27ef: Neither blocks nor log segments found. Creating new log.
Sep 29 03:15:19 dist-test-slave-nbjg krb5kdc[15337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:19.320674 15428 log.cc:824] T 00000000000000000000000000000000 P 1324724fd0e94195be72e9cedb9c27ef: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:19.322794 15428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1324724fd0e94195be72e9cedb9c27ef: No bootstrap required, opened a new log
I20260929 03:15:19.325766 15431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:19.312890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34946 (local address 127.13.238.190:37979)
0929 03:15:19.313348 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:19.313364 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:19.313405 (+ 41us) server_negotiation.cc:407] Connection header received
0929 03:15:19.314093 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:19.314111 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:19.314520 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:19.314843 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:19.315660 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:19.316639 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:19.317487 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:19.318007 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:19.318153 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:19.320762 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:19.320768 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:19.320782 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:19.320816 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:19.322959 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:19.323470 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:19.323475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:19.323480 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:19.323593 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:19.323872 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:19.323874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:19.323875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:19.324263 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:19.324420 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:19.324699 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 03:15:19.325000 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":128,"threads_started":1}
I20260929 03:15:19.326467 15428 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1324724fd0e94195be72e9cedb9c27ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1324724fd0e94195be72e9cedb9c27ef" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37979 } }
I20260929 03:15:19.326707 15428 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1324724fd0e94195be72e9cedb9c27ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:19.326790 15428 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1324724fd0e94195be72e9cedb9c27ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1324724fd0e94195be72e9cedb9c27ef, State: Initialized, Role: FOLLOWER
I20260929 03:15:19.327301 15428 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1324724fd0e94195be72e9cedb9c27ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1324724fd0e94195be72e9cedb9c27ef" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37979 } }
I20260929 03:15:19.327468 15428 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1324724fd0e94195be72e9cedb9c27ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:15:19.327531 15428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1324724fd0e94195be72e9cedb9c27ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:15:19.327626 15428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1324724fd0e94195be72e9cedb9c27ef [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:19.328660 15428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1324724fd0e94195be72e9cedb9c27ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1324724fd0e94195be72e9cedb9c27ef" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37979 } }
I20260929 03:15:19.328985 15428 leader_election.cc:302] T 00000000000000000000000000000000 P 1324724fd0e94195be72e9cedb9c27ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1324724fd0e94195be72e9cedb9c27ef; no voters:
I20260929 03:15:19.329269 15428 leader_election.cc:288] T 00000000000000000000000000000000 P 1324724fd0e94195be72e9cedb9c27ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:15:19.329397 15433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1324724fd0e94195be72e9cedb9c27ef [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:19.329622 15433 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1324724fd0e94195be72e9cedb9c27ef [term 1 LEADER]: Becoming Leader. State: Replica: 1324724fd0e94195be72e9cedb9c27ef, State: Running, Role: LEADER
I20260929 03:15:19.329974 15433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1324724fd0e94195be72e9cedb9c27ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1324724fd0e94195be72e9cedb9c27ef" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37979 } }
I20260929 03:15:19.330597 15428 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1324724fd0e94195be72e9cedb9c27ef [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:19.331691 15435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1324724fd0e94195be72e9cedb9c27ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1324724fd0e94195be72e9cedb9c27ef. Latest consensus state: current_term: 1 leader_uuid: "1324724fd0e94195be72e9cedb9c27ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1324724fd0e94195be72e9cedb9c27ef" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37979 } } }
I20260929 03:15:19.331825 15435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1324724fd0e94195be72e9cedb9c27ef [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:19.331969 15434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1324724fd0e94195be72e9cedb9c27ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1324724fd0e94195be72e9cedb9c27ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1324724fd0e94195be72e9cedb9c27ef" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37979 } } }
I20260929 03:15:19.332288 15434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1324724fd0e94195be72e9cedb9c27ef [sys.catalog]: This master's current role is: LEADER
W20260929 03:15:19.335242 15448 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1324724fd0e94195be72e9cedb9c27ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:15:19.335312 15448 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:15:19.335403 15443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:15:19.336349 15443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:15:19.342079 15443 catalog_manager.cc:1347] Generated new cluster ID: 1f4200022e2e43c89eed97e7bf7f1f52
I20260929 03:15:19.342131 15443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:15:19.350004 15443 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:15:19.351210 15443 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:15:19.358253 15443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1324724fd0e94195be72e9cedb9c27ef: Generated new TSK 0
I20260929 03:15:19.359036 15443 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
I20260929 03:15:19.394053 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:0
--local_ip_for_outbound_sockets=127.13.238.129
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:37979
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:45977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:19.507429 15454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:19.507709 15454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:19.507754 15454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:19.511934 15454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:19.511993 15454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:19.512035 15454 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:15:19.512089 15454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:15:19.517413 15454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:45977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.238.190:37979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:19.518567 15454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:19.519644 15454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:19.529731 15459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:19.529735 15460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:19.529861 15454 server_base.cc:1034] running on GCE node
W20260929 03:15:19.530095 15462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:19.530510 15454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:19.531258 15454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:19.532419 15454 hybrid_clock.cc:648] HybridClock initialized: now 1790651719532394 us; error 37 us; skew 500 ppm
Sep 29 03:15:19 dist-test-slave-nbjg krb5kdc[15337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651719, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:19.535161 15454 init.cc:377] Logged in from keytab as kudu/127.13.238.129@KRBTEST.COM (short username kudu)
I20260929 03:15:19.536026 15454 webserver.cc:460] Webserver started at http://127.13.238.129:34407/ using document root <none> and password file <none>
I20260929 03:15:19.536567 15454 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:19.536630 15454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:19.536815 15454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:19.538937 15454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-0/data/instance:
uuid: "f6d71a5c949f4e6cacb9479d6e65ee10"
format_stamp: "Formatted at 2026-09-29 03:15:19 on dist-test-slave-nbjg"
server_key: "8b8697eca0c7a7238af19e30c3a4785d"
server_key_iv: "c088873b8b39ebc688292f0147d23bb4"
server_key_version: "encryptionkey@0"
I20260929 03:15:19.539458 15454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance:
uuid: "f6d71a5c949f4e6cacb9479d6e65ee10"
format_stamp: "Formatted at 2026-09-29 03:15:19 on dist-test-slave-nbjg"
server_key: "8b8697eca0c7a7238af19e30c3a4785d"
server_key_iv: "c088873b8b39ebc688292f0147d23bb4"
server_key_version: "encryptionkey@0"
I20260929 03:15:19.543066 15454 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:15:19.545406 15469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:19.546657 15454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:19.546767 15454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "f6d71a5c949f4e6cacb9479d6e65ee10"
format_stamp: "Formatted at 2026-09-29 03:15:19 on dist-test-slave-nbjg"
server_key: "8b8697eca0c7a7238af19e30c3a4785d"
server_key_iv: "c088873b8b39ebc688292f0147d23bb4"
server_key_version: "encryptionkey@0"
I20260929 03:15:19.546864 15454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:19.573187 15454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:19.576450 15454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:19.576622 15454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:19.578102 15454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:19.578151 15454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:19.578231 15454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:19.578259 15454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:15:19 dist-test-slave-nbjg krb5kdc[15337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651719, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:19.588699 15454 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:33215
I20260929 03:15:19.588770 15584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:33215 every 8 connection(s)
I20260929 03:15:19.589557 15454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:15:19.590458 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 15454
I20260929 03:15:19.590585 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance
I20260929 03:15:19.590874 14266 external_mini_cluster.cc:1428] Setting key a1acbdc68aed8d09a0dbb41ae98e5277
I20260929 03:15:19.591185 15431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:19.578902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:43317 (local address 127.13.238.190:37979)
0929 03:15:19.579077 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:19.579082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:19.579999 (+ 917us) server_negotiation.cc:407] Connection header received
0929 03:15:19.581192 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:19.581196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:19.581258 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:19.581351 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:19.582986 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:19.583631 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:19.584762 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:19.585048 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:19.585142 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:19.587603 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:19.587607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:19.587610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:19.587639 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:19.589068 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:19.589526 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:19.589530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:19.589532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:19.589611 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:19.589931 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:19.589935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:19.589936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:19.590166 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:19.590319 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:19.590937 (+ 618us) server_negotiation.cc:299] Negotiation successful
0929 03:15:19.591065 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 03:15:19.592114 15478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:19.579324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:37979 (local address 127.13.238.129:43317)
0929 03:15:19.579816 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:19.579877 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:19.580950 (+ 1073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:19.581574 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:19.581584 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:19.582079 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:19.582798 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:19.582821 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:19.583759 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:19.583763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:19.584640 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:19.584650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:19.585209 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:19.585213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:19.585379 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:19.585906 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:19.585927 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:19.587443 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:19.589170 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:19.589176 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:19.589191 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:19.589427 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:19.589706 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:19.589709 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:19.589711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:19.589842 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:19.590364 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:19.590371 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:19.590667 (+ 296us) client_negotiation.cc:769] Sending connection context
0929 03:15:19.590872 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 03:15:19.591199 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":113,"threads_started":1}
I20260929 03:15:19.593292 15585 heartbeater.cc:347] Connected to a master server at 127.13.238.190:37979
I20260929 03:15:19.593436 15585 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:19.593988 15585 heartbeater.cc:511] Master 127.13.238.190:37979 requested a full tablet report, sending...
I20260929 03:15:19.595690 15382 ts_manager.cc:194] Registered new tserver with Master: f6d71a5c949f4e6cacb9479d6e65ee10 (127.13.238.129:33215)
I20260929 03:15:19.597079 15382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.129@KRBTEST.COM'} at 127.13.238.129:43317
WARNING: no policy specified for kudu/127.13.238.130@KRBTEST.COM; defaulting to no policy
I20260929 03:15:19.602908 15431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:19.595497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:43937 (local address 127.13.238.190:37979)
0929 03:15:19.595608 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:19.595612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:19.595789 (+ 177us) server_negotiation.cc:407] Connection header received
0929 03:15:19.595931 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:19.595934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:19.595984 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:19.596062 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:19.596522 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:19.597242 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:19.598486 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:19.598711 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:19.598804 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:19.599984 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:19.599987 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:19.599989 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:19.600011 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:19.601382 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:19.601824 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:19.601826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:19.601828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:19.601894 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:19.602149 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:19.602152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:19.602154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:19.602384 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:19.602462 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:19.602697 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 03:15:19.602791 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:15:19.602917 15592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:19.595417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:37979 (local address 127.13.238.129:43937)
0929 03:15:19.595715 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:19.595728 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:19.595849 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:19.596085 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:19.596088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:19.596280 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:19.596395 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:19.596402 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:19.597381 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:19.597385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:19.598361 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:19.598372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:19.598828 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:19.598832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:19.598917 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:19.599383 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:19.599397 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:19.599847 (+ 450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:19.601494 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:19.601497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:19.601499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:19.601734 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:19.601970 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:19.601973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:19.601975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:19.602065 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:19.602483 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:19.602486 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:19.602572 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 03:15:19.602653 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 03:15:19.602764 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":83,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
I20260929 03:15:19.620944 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:0
--local_ip_for_outbound_sockets=127.13.238.130
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:37979
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:45977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:19.731125 15594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:19.731406 15594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:19.731467 15594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:19.735457 15594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:19.735514 15594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:19.735550 15594 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:15:19.735591 15594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:15:19.740991 15594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:45977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.238.190:37979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:19.742125 15594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:19.743211 15594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:19.752879 15602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:19.752868 15600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:19.752866 15599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:19.753159 15594 server_base.cc:1034] running on GCE node
I20260929 03:15:19.753635 15594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:19.754319 15594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:19.755494 15594 hybrid_clock.cc:648] HybridClock initialized: now 1790651719755471 us; error 32 us; skew 500 ppm
Sep 29 03:15:19 dist-test-slave-nbjg krb5kdc[15337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651719, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:19.758129 15594 init.cc:377] Logged in from keytab as kudu/127.13.238.130@KRBTEST.COM (short username kudu)
I20260929 03:15:19.759043 15594 webserver.cc:460] Webserver started at http://127.13.238.130:34655/ using document root <none> and password file <none>
I20260929 03:15:19.759624 15594 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:19.759696 15594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:19.759902 15594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:19.762064 15594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-1/data/instance:
uuid: "88872278b28e4a6893cf11479a855e87"
format_stamp: "Formatted at 2026-09-29 03:15:19 on dist-test-slave-nbjg"
server_key: "f3671165efb50ecc90604d16fe234b6e"
server_key_iv: "86867adedb73439cf66405c577f51cad"
server_key_version: "encryptionkey@0"
I20260929 03:15:19.762684 15594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance:
uuid: "88872278b28e4a6893cf11479a855e87"
format_stamp: "Formatted at 2026-09-29 03:15:19 on dist-test-slave-nbjg"
server_key: "f3671165efb50ecc90604d16fe234b6e"
server_key_iv: "86867adedb73439cf66405c577f51cad"
server_key_version: "encryptionkey@0"
I20260929 03:15:19.767098 15594 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:15:19.769709 15609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:19.771008 15594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:19.771135 15594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "88872278b28e4a6893cf11479a855e87"
format_stamp: "Formatted at 2026-09-29 03:15:19 on dist-test-slave-nbjg"
server_key: "f3671165efb50ecc90604d16fe234b6e"
server_key_iv: "86867adedb73439cf66405c577f51cad"
server_key_version: "encryptionkey@0"
I20260929 03:15:19.771234 15594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:19.799029 15594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:19.802034 15594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:19.802232 15594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:19.803678 15594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:19.803725 15594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:19.803779 15594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:19.803820 15594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:15:19 dist-test-slave-nbjg krb5kdc[15337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651719, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:19.813529 15594 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:35591
I20260929 03:15:19.813598 15724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:35591 every 8 connection(s)
I20260929 03:15:19.814589 15594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:15:19.816902 15431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:19.804493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:48003 (local address 127.13.238.190:37979)
0929 03:15:19.804639 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:19.804643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:19.805514 (+ 871us) server_negotiation.cc:407] Connection header received
0929 03:15:19.806648 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:19.806651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:19.806707 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:19.806785 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:19.807983 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:19.808579 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:19.810288 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:19.810546 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:19.810639 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:19.813501 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:19.813505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:19.813508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:19.813536 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:19.814963 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:19.815548 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:19.815558 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:19.815560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:19.815645 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:19.815960 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:19.815963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:19.815965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:19.816140 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:19.816215 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:19.816680 (+ 465us) server_negotiation.cc:299] Negotiation successful
0929 03:15:19.816785 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 03:15:19.817385 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 15594
I20260929 03:15:19.817473 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance
I20260929 03:15:19.817700 14266 external_mini_cluster.cc:1428] Setting key d94d3b4fc59f24e6ba4a673cd4096144
I20260929 03:15:19.817806 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:19.804881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:37979 (local address 127.13.238.130:48003)
0929 03:15:19.805344 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:19.805394 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:19.806380 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:19.806968 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:19.806976 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:19.807287 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:19.807838 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:19.807853 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:19.808692 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:19.808696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:19.810145 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:19.810160 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:19.810675 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:19.810679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:19.810885 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:19.811427 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:19.811448 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:19.813335 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:19.815089 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:19.815094 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:19.815108 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:19.815422 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:19.815721 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:19.815724 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:19.815725 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:19.815893 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:19.816253 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:19.816259 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:19.816486 (+ 227us) client_negotiation.cc:769] Sending connection context
0929 03:15:19.816649 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 03:15:19.816901 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":111,"threads_started":1}
I20260929 03:15:19.821144 15725 heartbeater.cc:347] Connected to a master server at 127.13.238.190:37979
I20260929 03:15:19.821271 15725 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:19.821800 15725 heartbeater.cc:511] Master 127.13.238.190:37979 requested a full tablet report, sending...
I20260929 03:15:19.822796 15382 ts_manager.cc:194] Registered new tserver with Master: 88872278b28e4a6893cf11479a855e87 (127.13.238.130:35591)
I20260929 03:15:19.823416 15382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.130@KRBTEST.COM'} at 127.13.238.130:48003
WARNING: no policy specified for kudu/127.13.238.131@KRBTEST.COM; defaulting to no policy
I20260929 03:15:19.829550 15431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:19.821428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:50533 (local address 127.13.238.190:37979)
0929 03:15:19.821540 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:19.821544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:19.822459 (+ 915us) server_negotiation.cc:407] Connection header received
0929 03:15:19.822550 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:19.822553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:19.822596 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:19.822670 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:19.823137 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:19.823918 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:19.825083 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:19.825406 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:19.825491 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:19.826722 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:19.826725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:19.826727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:19.826748 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:19.827987 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:19.828460 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:19.828463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:19.828464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:19.828545 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:19.828812 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:19.828815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:19.828817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:19.829024 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:19.829105 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:19.829336 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 03:15:19.829442 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:15:19.829550 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:19.821349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:37979 (local address 127.13.238.130:50533)
0929 03:15:19.822366 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:19.822380 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:19.822478 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:19.822703 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:19.822706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:19.822903 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:19.823016 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:19.823023 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:19.824065 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:19.824069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:19.824951 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:19.824962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:19.825637 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:19.825640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:19.825729 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:19.826161 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:19.826175 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:19.826598 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:19.828084 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:19.828087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:19.828089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:19.828288 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:19.828652 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:19.828655 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:19.828656 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:19.828731 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:19.829102 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:19.829104 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:19.829176 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 03:15:19.829267 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 03:15:19.829378 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":935,"thread_start_us":210,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
I20260929 03:15:19.845542 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:0
--local_ip_for_outbound_sockets=127.13.238.131
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:37979
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:45977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:19.956898 15734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:19.957144 15734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:19.957185 15734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:19.961184 15734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:19.961237 15734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:19.961277 15734 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:15:19.961313 15734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:15:19.966506 15734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:45977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.238.190:37979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:19.967545 15734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:19.968555 15734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:19.978461 15742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:19.978479 15739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:19.978603 15740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:19.978740 15734 server_base.cc:1034] running on GCE node
I20260929 03:15:19.979236 15734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:19.980002 15734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:19.981201 15734 hybrid_clock.cc:648] HybridClock initialized: now 1790651719981174 us; error 44 us; skew 500 ppm
Sep 29 03:15:19 dist-test-slave-nbjg krb5kdc[15337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651719, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:19.984344 15734 init.cc:377] Logged in from keytab as kudu/127.13.238.131@KRBTEST.COM (short username kudu)
I20260929 03:15:19.985325 15734 webserver.cc:460] Webserver started at http://127.13.238.131:39891/ using document root <none> and password file <none>
I20260929 03:15:19.985917 15734 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:19.985986 15734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:19.986166 15734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:19.988417 15734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-2/data/instance:
uuid: "63033321bdae433e9119b8e602eb9bcd"
format_stamp: "Formatted at 2026-09-29 03:15:19 on dist-test-slave-nbjg"
server_key: "e4b56929fb7af559d36dcdf7f738f335"
server_key_iv: "c2a0bdb3ac72d28d55edd6f772052352"
server_key_version: "encryptionkey@0"
I20260929 03:15:19.988989 15734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance:
uuid: "63033321bdae433e9119b8e602eb9bcd"
format_stamp: "Formatted at 2026-09-29 03:15:19 on dist-test-slave-nbjg"
server_key: "e4b56929fb7af559d36dcdf7f738f335"
server_key_iv: "c2a0bdb3ac72d28d55edd6f772052352"
server_key_version: "encryptionkey@0"
I20260929 03:15:19.993016 15734 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 03:15:19.995983 15749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:19.997398 15734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:15:19.997512 15734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "63033321bdae433e9119b8e602eb9bcd"
format_stamp: "Formatted at 2026-09-29 03:15:19 on dist-test-slave-nbjg"
server_key: "e4b56929fb7af559d36dcdf7f738f335"
server_key_iv: "c2a0bdb3ac72d28d55edd6f772052352"
server_key_version: "encryptionkey@0"
I20260929 03:15:19.997613 15734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:20.007277 15734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:20.010113 15734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:20.010321 15734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:20.011771 15734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:20.011819 15734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:20.011873 15734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:20.011900 15734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:15:20 dist-test-slave-nbjg krb5kdc[15337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651719, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:20.021891 15734 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:36541
I20260929 03:15:20.022351 15864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:36541 every 8 connection(s)
I20260929 03:15:20.022992 15734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:15:20.025983 15431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:20.012576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:43959 (local address 127.13.238.190:37979)
0929 03:15:20.012729 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:20.012735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:20.013630 (+ 895us) server_negotiation.cc:407] Connection header received
0929 03:15:20.014754 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:20.014759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:20.014824 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:20.014880 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:20.016389 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.016986 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.018245 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.018476 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.018575 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:20.022237 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:20.022243 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:20.022246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:20.022292 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:20.023908 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:20.024443 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:20.024447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:20.024449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:20.024536 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:20.024862 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:20.024866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:20.024867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:20.025073 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:20.025164 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:20.025677 (+ 513us) server_negotiation.cc:299] Negotiation successful
0929 03:15:20.025808 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 03:15:20.026805 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:20.013002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:37979 (local address 127.13.238.131:43959)
0929 03:15:20.013465 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:20.013514 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:20.014488 (+ 974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:20.015222 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:20.015231 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:20.015585 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:20.016228 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:20.016246 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.017117 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.017120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:20.018100 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:20.018112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.018663 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.018668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:20.018834 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:20.020030 (+ 1196us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:20.020051 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:20.022002 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:20.024054 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:20.024060 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:20.024076 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:20.024335 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:20.024627 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:20.024630 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:20.024632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:20.024769 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:20.025179 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:20.025186 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:20.025431 (+ 245us) client_negotiation.cc:769] Sending connection context
0929 03:15:20.025608 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 03:15:20.025935 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":109,"threads_started":1}
I20260929 03:15:20.027984 15865 heartbeater.cc:347] Connected to a master server at 127.13.238.190:37979
I20260929 03:15:20.028121 15865 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:20.028610 15865 heartbeater.cc:511] Master 127.13.238.190:37979 requested a full tablet report, sending...
I20260929 03:15:20.029633 15379 ts_manager.cc:194] Registered new tserver with Master: 63033321bdae433e9119b8e602eb9bcd (127.13.238.131:36541)
I20260929 03:15:20.030400 15379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.131@KRBTEST.COM'} at 127.13.238.131:43959
I20260929 03:15:20.031986 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 15734
I20260929 03:15:20.032101 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance
I20260929 03:15:20.032370 14266 external_mini_cluster.cc:1428] Setting key ce9f4303d150df73f947e7dddd12d91f
I20260929 03:15:20.034495 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:15:20.036787 15431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:20.030129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:46667 (local address 127.13.238.190:37979)
0929 03:15:20.030270 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:20.030273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:20.030376 (+ 103us) server_negotiation.cc:407] Connection header received
0929 03:15:20.030471 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:20.030474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:20.030519 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:20.030584 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:20.030978 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.031544 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.032612 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.032930 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.033024 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:20.034047 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:20.034051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:20.034054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:20.034076 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:20.035307 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:20.035731 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:20.035734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:20.035736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:20.035811 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:20.036056 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:20.036059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:20.036061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:20.036221 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:20.036305 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:20.036554 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 03:15:20.036653 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:15:20.036798 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:20.030003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:37979 (local address 127.13.238.131:46667)
0929 03:15:20.030292 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:20.030305 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:20.030399 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:20.030604 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:20.030607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:20.030770 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:20.030860 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:20.030865 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.031661 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.031664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:20.032511 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:20.032522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.033040 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.033043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:20.033134 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:20.033615 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:20.033630 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:20.033914 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:20.035415 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:20.035418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:20.035420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:20.035602 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:20.035894 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:20.035897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:20.035900 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:20.035977 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:20.036310 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:20.036313 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:20.036405 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 03:15:20.036487 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 03:15:20.036618 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":96,"threads_started":1}
Sep 29 03:15:20 dist-test-slave-nbjg krb5kdc[15337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651720, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:15:20 dist-test-slave-nbjg krb5kdc[15337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651720, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:20.058658 15431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:20.051155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34948 (local address 127.13.238.190:37979)
0929 03:15:20.051297 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:20.051301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:20.051313 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:15:20.051353 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:20.051356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:20.051394 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:20.051461 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:20.052183 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.052685 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.053492 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.053698 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.053767 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:20.056041 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:20.056044 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:20.056046 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:20.056069 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:20.057156 (+ 1087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:20.057492 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:20.057494 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:20.057495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:20.057559 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:20.057834 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:20.057838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:20.057839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:20.058041 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:20.058148 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:20.058405 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 03:15:20.058502 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 03:15:20.061487 15379 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:15:20.064503 15379 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:15:20.078691 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:20.072666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:33215 (local address 127.0.0.1:40030)
0929 03:15:20.073205 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:20.073218 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:20.073293 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:20.074431 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:20.074441 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:20.074483 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:20.074797 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:20.074802 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.076202 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.076206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:20.077141 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:20.077150 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.078240 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.078244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:20.078351 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:20.078385 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 03:15:20.078478 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 03:15:20.078568 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":208,"spinlock_wait_cycles":7680,"thread_start_us":96,"threads_started":1}
I20260929 03:15:20.078778 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:20.073620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40030 (local address 127.13.238.129:33215)
0929 03:15:20.073958 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:20.073970 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:20.073989 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:15:20.074107 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:20.074112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:20.074280 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:20.074907 (+ 627us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:20.075027 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.076087 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.077259 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.078090 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.078248 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:20.078383 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:20.078535 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 03:15:20.078627 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":77,"threads_started":1}
I20260929 03:15:20.079486 15883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:20.072558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:35591 (local address 127.0.0.1:56284)
0929 03:15:20.073007 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:20.073053 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:20.073170 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:20.074431 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:20.074441 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:20.074477 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:20.074631 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:20.074637 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.075911 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.075914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:20.077113 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:20.077124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.078998 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.079002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:20.079091 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:20.079114 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:15:20.079196 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 03:15:20.079266 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":26,"spinlock_wait_cycles":14976,"thread_start_us":79,"threads_started":1}
I20260929 03:15:20.079631 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:20.072726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56284 (local address 127.13.238.130:35591)
0929 03:15:20.073869 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:20.073881 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:20.073904 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:15:20.074010 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:20.074016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:20.074281 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:20.074581 (+ 300us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:20.074711 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.075803 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.077235 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.078090 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.078248 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:20.078383 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:20.079240 (+ 857us) server_negotiation.cc:299] Negotiation successful
0929 03:15:20.079396 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1039,"thread_start_us":91,"threads_started":1}
I20260929 03:15:20.080447 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:20.073275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:36541 (local address 127.0.0.1:32778)
0929 03:15:20.073535 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:20.073546 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:20.073623 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:20.074431 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:20.074437 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:20.074469 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:20.074608 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:20.074615 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.075682 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.075685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:20.077123 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:20.077129 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.080073 (+ 2944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.080076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:20.080161 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:20.080183 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 03:15:20.080247 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 03:15:20.080315 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":22,"spinlock_wait_cycles":3584,"thread_start_us":77,"threads_started":1}
I20260929 03:15:20.080487 15886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:20.073600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32778 (local address 127.13.238.131:36541)
0929 03:15:20.073895 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:20.073906 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:20.073928 (+ 22us) server_negotiation.cc:407] Connection header received
0929 03:15:20.074052 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:20.074058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:20.074287 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:20.074430 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:20.074714 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.075585 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.078449 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.079240 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.080101 (+ 861us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:20.080226 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:20.080271 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 03:15:20.080352 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":92,"threads_started":1}
I20260929 03:15:20.082026 15519 tablet_service.cc:1467] Processing CreateTablet for tablet d7550ddcea624627bf83f1d7f323d3ff (DEFAULT_TABLE table=test-table [id=4b2067bd31a14c819a7843fb48931cac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:20.082362 15659 tablet_service.cc:1467] Processing CreateTablet for tablet d7550ddcea624627bf83f1d7f323d3ff (DEFAULT_TABLE table=test-table [id=4b2067bd31a14c819a7843fb48931cac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:20.083235 15519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7550ddcea624627bf83f1d7f323d3ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:20.083287 15659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7550ddcea624627bf83f1d7f323d3ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:20.083307 15799 tablet_service.cc:1467] Processing CreateTablet for tablet d7550ddcea624627bf83f1d7f323d3ff (DEFAULT_TABLE table=test-table [id=4b2067bd31a14c819a7843fb48931cac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:20.084236 15799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7550ddcea624627bf83f1d7f323d3ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:20.089825 15888 tablet_bootstrap.cc:492] T d7550ddcea624627bf83f1d7f323d3ff P 88872278b28e4a6893cf11479a855e87: Bootstrap starting.
I20260929 03:15:20.091203 15889 tablet_bootstrap.cc:492] T d7550ddcea624627bf83f1d7f323d3ff P f6d71a5c949f4e6cacb9479d6e65ee10: Bootstrap starting.
I20260929 03:15:20.091766 15890 tablet_bootstrap.cc:492] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd: Bootstrap starting.
I20260929 03:15:20.092262 15888 tablet_bootstrap.cc:654] T d7550ddcea624627bf83f1d7f323d3ff P 88872278b28e4a6893cf11479a855e87: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:20.093209 15888 log.cc:824] T d7550ddcea624627bf83f1d7f323d3ff P 88872278b28e4a6893cf11479a855e87: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:20.093309 15889 tablet_bootstrap.cc:654] T d7550ddcea624627bf83f1d7f323d3ff P f6d71a5c949f4e6cacb9479d6e65ee10: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:20.093516 15890 tablet_bootstrap.cc:654] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:20.094071 15889 log.cc:824] T d7550ddcea624627bf83f1d7f323d3ff P f6d71a5c949f4e6cacb9479d6e65ee10: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:20.094293 15890 log.cc:824] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:20.095789 15889 tablet_bootstrap.cc:492] T d7550ddcea624627bf83f1d7f323d3ff P f6d71a5c949f4e6cacb9479d6e65ee10: No bootstrap required, opened a new log
I20260929 03:15:20.095850 15888 tablet_bootstrap.cc:492] T d7550ddcea624627bf83f1d7f323d3ff P 88872278b28e4a6893cf11479a855e87: No bootstrap required, opened a new log
I20260929 03:15:20.095975 15889 ts_tablet_manager.cc:1397] T d7550ddcea624627bf83f1d7f323d3ff P f6d71a5c949f4e6cacb9479d6e65ee10: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 03:15:20.096037 15888 ts_tablet_manager.cc:1397] T d7550ddcea624627bf83f1d7f323d3ff P 88872278b28e4a6893cf11479a855e87: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 03:15:20.096098 15890 tablet_bootstrap.cc:492] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd: No bootstrap required, opened a new log
I20260929 03:15:20.096247 15890 ts_tablet_manager.cc:1397] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 03:15:20.098868 15889 raft_consensus.cc:348] T d7550ddcea624627bf83f1d7f323d3ff P f6d71a5c949f4e6cacb9479d6e65ee10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:20.099077 15889 raft_consensus.cc:374] T d7550ddcea624627bf83f1d7f323d3ff P f6d71a5c949f4e6cacb9479d6e65ee10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:20.099126 15889 raft_consensus.cc:729] T d7550ddcea624627bf83f1d7f323d3ff P f6d71a5c949f4e6cacb9479d6e65ee10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d71a5c949f4e6cacb9479d6e65ee10, State: Initialized, Role: FOLLOWER
I20260929 03:15:20.099536 15889 consensus_queue.cc:260] T d7550ddcea624627bf83f1d7f323d3ff P f6d71a5c949f4e6cacb9479d6e65ee10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:20.099797 15890 raft_consensus.cc:348] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:20.099797 15888 raft_consensus.cc:348] T d7550ddcea624627bf83f1d7f323d3ff P 88872278b28e4a6893cf11479a855e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:20.100033 15890 raft_consensus.cc:374] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:20.100034 15888 raft_consensus.cc:374] T d7550ddcea624627bf83f1d7f323d3ff P 88872278b28e4a6893cf11479a855e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:20.100088 15890 raft_consensus.cc:729] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63033321bdae433e9119b8e602eb9bcd, State: Initialized, Role: FOLLOWER
I20260929 03:15:20.100097 15888 raft_consensus.cc:729] T d7550ddcea624627bf83f1d7f323d3ff P 88872278b28e4a6893cf11479a855e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88872278b28e4a6893cf11479a855e87, State: Initialized, Role: FOLLOWER
I20260929 03:15:20.100299 15585 heartbeater.cc:503] Master 127.13.238.190:37979 was elected leader, sending a full tablet report...
I20260929 03:15:20.100579 15889 ts_tablet_manager.cc:1428] T d7550ddcea624627bf83f1d7f323d3ff P f6d71a5c949f4e6cacb9479d6e65ee10: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 03:15:20.100528 15890 consensus_queue.cc:260] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:20.100983 15888 consensus_queue.cc:260] T d7550ddcea624627bf83f1d7f323d3ff P 88872278b28e4a6893cf11479a855e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:20.101471 15890 ts_tablet_manager.cc:1428] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 03:15:20.101795 15865 heartbeater.cc:503] Master 127.13.238.190:37979 was elected leader, sending a full tablet report...
I20260929 03:15:20.102300 15725 heartbeater.cc:503] Master 127.13.238.190:37979 was elected leader, sending a full tablet report...
I20260929 03:15:20.102756 15888 ts_tablet_manager.cc:1428] T d7550ddcea624627bf83f1d7f323d3ff P 88872278b28e4a6893cf11479a855e87: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260929 03:15:20.274097 15866 tablet.cc:2369] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:15:20.277264 15895 raft_consensus.cc:482] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:15:20.277456 15895 raft_consensus.cc:504] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:20.278748 15895 leader_election.cc:288] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88872278b28e4a6893cf11479a855e87 (127.13.238.130:35591), f6d71a5c949f4e6cacb9479d6e65ee10 (127.13.238.129:33215)
I20260929 03:15:20.282653 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:20.279042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:33215 (local address 127.13.238.131:60263)
0929 03:15:20.279171 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:20.279186 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:20.279295 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:20.279575 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:20.279578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:20.279599 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:20.279720 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:20.279733 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.280508 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.280511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:20.281560 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:20.281571 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.282318 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.282321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:20.282411 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:20.282432 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 03:15:20.282490 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 03:15:20.282545 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 03:15:20.282711 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:20.279128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:60263 (local address 127.13.238.129:33215)
0929 03:15:20.279251 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:20.279255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:20.279272 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:15:20.279410 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:20.279414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:20.279466 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:20.279531 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:20.279837 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.280398 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.281690 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.282173 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.282324 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:20.282377 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:20.282519 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 03:15:20.282587 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:15:20.283303 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:20.279121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:39605 (local address 127.13.238.130:35591)
0929 03:15:20.279231 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:20.279236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:20.279611 (+ 375us) server_negotiation.cc:407] Connection header received
0929 03:15:20.279687 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:20.279689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:20.279729 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:20.279787 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:20.280263 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.280842 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.282074 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.282627 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.282765 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:20.282813 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:20.283137 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 03:15:20.283200 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:15:20.283326 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:20.279045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:35591 (local address 127.13.238.131:39605)
0929 03:15:20.279517 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:20.279529 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:20.279612 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:20.279809 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:20.279812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:20.279828 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:20.280164 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:20.280170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.280947 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.280951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:20.281961 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:20.281969 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.282910 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.282914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:20.283011 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:20.283034 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:15:20.283107 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 03:15:20.283171 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":127,"thread_start_us":142,"threads_started":1}
I20260929 03:15:20.283381 15539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7550ddcea624627bf83f1d7f323d3ff" candidate_uuid: "63033321bdae433e9119b8e602eb9bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" is_pre_election: true
I20260929 03:15:20.283717 15539 raft_consensus.cc:2454] T d7550ddcea624627bf83f1d7f323d3ff P f6d71a5c949f4e6cacb9479d6e65ee10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63033321bdae433e9119b8e602eb9bcd in term 0.
I20260929 03:15:20.284092 15679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7550ddcea624627bf83f1d7f323d3ff" candidate_uuid: "63033321bdae433e9119b8e602eb9bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88872278b28e4a6893cf11479a855e87" is_pre_election: true
I20260929 03:15:20.284227 15750 leader_election.cc:302] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63033321bdae433e9119b8e602eb9bcd, f6d71a5c949f4e6cacb9479d6e65ee10; no voters:
I20260929 03:15:20.284399 15679 raft_consensus.cc:2454] T d7550ddcea624627bf83f1d7f323d3ff P 88872278b28e4a6893cf11479a855e87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63033321bdae433e9119b8e602eb9bcd in term 0.
I20260929 03:15:20.284504 15895 raft_consensus.cc:2781] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:15:20.284596 15895 raft_consensus.cc:482] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:15:20.284646 15895 raft_consensus.cc:3037] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:20.285710 15895 raft_consensus.cc:504] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:20.286065 15895 leader_election.cc:288] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd [CANDIDATE]: Term 1 election: Requested vote from peers 88872278b28e4a6893cf11479a855e87 (127.13.238.130:35591), f6d71a5c949f4e6cacb9479d6e65ee10 (127.13.238.129:33215)
I20260929 03:15:20.286422 15679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7550ddcea624627bf83f1d7f323d3ff" candidate_uuid: "63033321bdae433e9119b8e602eb9bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88872278b28e4a6893cf11479a855e87"
I20260929 03:15:20.286439 15539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7550ddcea624627bf83f1d7f323d3ff" candidate_uuid: "63033321bdae433e9119b8e602eb9bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10"
I20260929 03:15:20.286532 15679 raft_consensus.cc:3037] T d7550ddcea624627bf83f1d7f323d3ff P 88872278b28e4a6893cf11479a855e87 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:20.286545 15539 raft_consensus.cc:3037] T d7550ddcea624627bf83f1d7f323d3ff P f6d71a5c949f4e6cacb9479d6e65ee10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:20.287526 15539 raft_consensus.cc:2454] T d7550ddcea624627bf83f1d7f323d3ff P f6d71a5c949f4e6cacb9479d6e65ee10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63033321bdae433e9119b8e602eb9bcd in term 1.
I20260929 03:15:20.287576 15679 raft_consensus.cc:2454] T d7550ddcea624627bf83f1d7f323d3ff P 88872278b28e4a6893cf11479a855e87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63033321bdae433e9119b8e602eb9bcd in term 1.
I20260929 03:15:20.287817 15750 leader_election.cc:302] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63033321bdae433e9119b8e602eb9bcd, f6d71a5c949f4e6cacb9479d6e65ee10; no voters:
I20260929 03:15:20.288038 15895 raft_consensus.cc:2781] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:20.288235 15895 raft_consensus.cc:686] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd [term 1 LEADER]: Becoming Leader. State: Replica: 63033321bdae433e9119b8e602eb9bcd, State: Running, Role: LEADER
I20260929 03:15:20.288545 15895 consensus_queue.cc:237] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:20.292001 15381 catalog_manager.cc:5495] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd reported cstate change: term changed from 0 to 1, leader changed from <none> to 63033321bdae433e9119b8e602eb9bcd (127.13.238.131). New cstate: current_term: 1 leader_uuid: "63033321bdae433e9119b8e602eb9bcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } health_report { overall_health: HEALTHY } } }
W20260929 03:15:20.316059 15726 tablet.cc:2369] T d7550ddcea624627bf83f1d7f323d3ff P 88872278b28e4a6893cf11479a855e87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:15:20.319126 15886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:20.314763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32784 (local address 127.13.238.131:36541)
0929 03:15:20.314905 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:20.314910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:20.314924 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:15:20.315103 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:20.315106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:20.315163 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:20.315245 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:20.315514 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.316219 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.317505 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.317726 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.317836 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:20.318259 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:20.318717 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:20.318806 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:20.318914 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 03:15:20.318971 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 03:15:20.324824 15679 raft_consensus.cc:1261] T d7550ddcea624627bf83f1d7f323d3ff P 88872278b28e4a6893cf11479a855e87 [term 1 FOLLOWER]: Refusing update from remote peer 63033321bdae433e9119b8e602eb9bcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:20.324824 15539 raft_consensus.cc:1261] T d7550ddcea624627bf83f1d7f323d3ff P f6d71a5c949f4e6cacb9479d6e65ee10 [term 1 FOLLOWER]: Refusing update from remote peer 63033321bdae433e9119b8e602eb9bcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:20.325496 15898 consensus_queue.cc:1035] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:20.325677 15895 consensus_queue.cc:1035] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:20.333662 15901 mvcc.cc:205] Tried to move back new op lower bound from 7334509446443646976 to 7334509446304591872. Current Snapshot: MvccSnapshot[applied={T|T < 7334509446443646976}]
I20260929 03:15:20.345921 15379 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:20.346120 15379 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:20.348798 15379 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1324724fd0e94195be72e9cedb9c27ef: Sending DeleteTablet for 3 replicas of tablet d7550ddcea624627bf83f1d7f323d3ff
I20260929 03:15:20.349840 15659 tablet_service.cc:1514] Processing DeleteTablet for tablet d7550ddcea624627bf83f1d7f323d3ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:20 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:56284
I20260929 03:15:20.349870 15519 tablet_service.cc:1514] Processing DeleteTablet for tablet d7550ddcea624627bf83f1d7f323d3ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:20 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:40030
I20260929 03:15:20.349958 15799 tablet_service.cc:1514] Processing DeleteTablet for tablet d7550ddcea624627bf83f1d7f323d3ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:20 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:32778
I20260929 03:15:20.350555 15911 tablet_replica.cc:330] T d7550ddcea624627bf83f1d7f323d3ff P 88872278b28e4a6893cf11479a855e87: stopping tablet replica
I20260929 03:15:20.350574 15379 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 03:15:20.350951 15911 raft_consensus.cc:2229] T d7550ddcea624627bf83f1d7f323d3ff P 88872278b28e4a6893cf11479a855e87 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260929 03:15:20.351027 15379 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:15:20.351269 15911 raft_consensus.cc:2258] T d7550ddcea624627bf83f1d7f323d3ff P 88872278b28e4a6893cf11479a855e87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:20.351701 15913 tablet_replica.cc:330] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd: stopping tablet replica
I20260929 03:15:20.351903 15912 tablet_replica.cc:330] T d7550ddcea624627bf83f1d7f323d3ff P f6d71a5c949f4e6cacb9479d6e65ee10: stopping tablet replica
I20260929 03:15:20.352020 15913 raft_consensus.cc:2229] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:15:20.352187 15912 raft_consensus.cc:2229] T d7550ddcea624627bf83f1d7f323d3ff P f6d71a5c949f4e6cacb9479d6e65ee10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:15:20.352376 15913 raft_consensus.cc:2258] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:20.352491 15912 raft_consensus.cc:2258] T d7550ddcea624627bf83f1d7f323d3ff P f6d71a5c949f4e6cacb9479d6e65ee10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:20.353952 15912 ts_tablet_manager.cc:1905] T d7550ddcea624627bf83f1d7f323d3ff P f6d71a5c949f4e6cacb9479d6e65ee10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:20.354012 15913 ts_tablet_manager.cc:1905] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:20.354945 15911 ts_tablet_manager.cc:1905] T d7550ddcea624627bf83f1d7f323d3ff P 88872278b28e4a6893cf11479a855e87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:20.356336 15659 tablet_service.cc:1467] Processing CreateTablet for tablet 517c4afdc55545929c070f0bcf0a16f4 (DEFAULT_TABLE table=test-table [id=bdc00811df0b4165aadae2777c2deeb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:20.356597 15659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 517c4afdc55545929c070f0bcf0a16f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:20.356611 15519 tablet_service.cc:1467] Processing CreateTablet for tablet 517c4afdc55545929c070f0bcf0a16f4 (DEFAULT_TABLE table=test-table [id=bdc00811df0b4165aadae2777c2deeb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:20.356786 15912 ts_tablet_manager.cc:1918] T d7550ddcea624627bf83f1d7f323d3ff P f6d71a5c949f4e6cacb9479d6e65ee10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:15:20.356823 15519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 517c4afdc55545929c070f0bcf0a16f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:20.356876 15912 log.cc:1196] T d7550ddcea624627bf83f1d7f323d3ff P f6d71a5c949f4e6cacb9479d6e65ee10: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-0/wal/wals/d7550ddcea624627bf83f1d7f323d3ff
I20260929 03:15:20.357167 15912 ts_tablet_manager.cc:1939] T d7550ddcea624627bf83f1d7f323d3ff P f6d71a5c949f4e6cacb9479d6e65ee10: Deleting consensus metadata
I20260929 03:15:20.358525 15366 catalog_manager.cc:4841] TS f6d71a5c949f4e6cacb9479d6e65ee10 (127.13.238.129:33215): tablet d7550ddcea624627bf83f1d7f323d3ff (table test-table [id=4b2067bd31a14c819a7843fb48931cac]) successfully deleted
I20260929 03:15:20.358850 15888 tablet_bootstrap.cc:492] T 517c4afdc55545929c070f0bcf0a16f4 P 88872278b28e4a6893cf11479a855e87: Bootstrap starting.
I20260929 03:15:20.358971 15889 tablet_bootstrap.cc:492] T 517c4afdc55545929c070f0bcf0a16f4 P f6d71a5c949f4e6cacb9479d6e65ee10: Bootstrap starting.
I20260929 03:15:20.359045 15799 tablet_service.cc:1467] Processing CreateTablet for tablet 517c4afdc55545929c070f0bcf0a16f4 (DEFAULT_TABLE table=test-table [id=bdc00811df0b4165aadae2777c2deeb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:20.359104 15911 ts_tablet_manager.cc:1918] T d7550ddcea624627bf83f1d7f323d3ff P 88872278b28e4a6893cf11479a855e87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:15:20.359184 15911 log.cc:1196] T d7550ddcea624627bf83f1d7f323d3ff P 88872278b28e4a6893cf11479a855e87: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-1/wal/wals/d7550ddcea624627bf83f1d7f323d3ff
I20260929 03:15:20.359472 15911 ts_tablet_manager.cc:1939] T d7550ddcea624627bf83f1d7f323d3ff P 88872278b28e4a6893cf11479a855e87: Deleting consensus metadata
I20260929 03:15:20.359690 15799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 517c4afdc55545929c070f0bcf0a16f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:20.359844 15888 tablet_bootstrap.cc:654] T 517c4afdc55545929c070f0bcf0a16f4 P 88872278b28e4a6893cf11479a855e87: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:20.359864 15889 tablet_bootstrap.cc:654] T 517c4afdc55545929c070f0bcf0a16f4 P f6d71a5c949f4e6cacb9479d6e65ee10: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:20.361279 15889 tablet_bootstrap.cc:492] T 517c4afdc55545929c070f0bcf0a16f4 P f6d71a5c949f4e6cacb9479d6e65ee10: No bootstrap required, opened a new log
I20260929 03:15:20.361356 15889 ts_tablet_manager.cc:1397] T 517c4afdc55545929c070f0bcf0a16f4 P f6d71a5c949f4e6cacb9479d6e65ee10: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:15:20.361533 15367 catalog_manager.cc:4841] TS 88872278b28e4a6893cf11479a855e87 (127.13.238.130:35591): tablet d7550ddcea624627bf83f1d7f323d3ff (table test-table [id=4b2067bd31a14c819a7843fb48931cac]) successfully deleted
I20260929 03:15:20.361773 15889 raft_consensus.cc:348] T 517c4afdc55545929c070f0bcf0a16f4 P f6d71a5c949f4e6cacb9479d6e65ee10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:20.361872 15889 raft_consensus.cc:374] T 517c4afdc55545929c070f0bcf0a16f4 P f6d71a5c949f4e6cacb9479d6e65ee10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:20.361914 15889 raft_consensus.cc:729] T 517c4afdc55545929c070f0bcf0a16f4 P f6d71a5c949f4e6cacb9479d6e65ee10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d71a5c949f4e6cacb9479d6e65ee10, State: Initialized, Role: FOLLOWER
I20260929 03:15:20.361940 15890 tablet_bootstrap.cc:492] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd: Bootstrap starting.
I20260929 03:15:20.362027 15889 consensus_queue.cc:260] T 517c4afdc55545929c070f0bcf0a16f4 P f6d71a5c949f4e6cacb9479d6e65ee10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:20.362311 15889 ts_tablet_manager.cc:1428] T 517c4afdc55545929c070f0bcf0a16f4 P f6d71a5c949f4e6cacb9479d6e65ee10: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:20.362869 15890 tablet_bootstrap.cc:654] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:20.363466 15888 tablet_bootstrap.cc:492] T 517c4afdc55545929c070f0bcf0a16f4 P 88872278b28e4a6893cf11479a855e87: No bootstrap required, opened a new log
I20260929 03:15:20.363549 15888 ts_tablet_manager.cc:1397] T 517c4afdc55545929c070f0bcf0a16f4 P 88872278b28e4a6893cf11479a855e87: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 03:15:20.363955 15888 raft_consensus.cc:348] T 517c4afdc55545929c070f0bcf0a16f4 P 88872278b28e4a6893cf11479a855e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:20.364058 15888 raft_consensus.cc:374] T 517c4afdc55545929c070f0bcf0a16f4 P 88872278b28e4a6893cf11479a855e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:20.364091 15888 raft_consensus.cc:729] T 517c4afdc55545929c070f0bcf0a16f4 P 88872278b28e4a6893cf11479a855e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88872278b28e4a6893cf11479a855e87, State: Initialized, Role: FOLLOWER
I20260929 03:15:20.364210 15888 consensus_queue.cc:260] T 517c4afdc55545929c070f0bcf0a16f4 P 88872278b28e4a6893cf11479a855e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:20.364313 15913 ts_tablet_manager.cc:1918] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:15:20.364401 15913 log.cc:1196] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/d7550ddcea624627bf83f1d7f323d3ff
I20260929 03:15:20.364477 15888 ts_tablet_manager.cc:1428] T 517c4afdc55545929c070f0bcf0a16f4 P 88872278b28e4a6893cf11479a855e87: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 03:15:20.364692 15913 ts_tablet_manager.cc:1939] T d7550ddcea624627bf83f1d7f323d3ff P 63033321bdae433e9119b8e602eb9bcd: Deleting consensus metadata
I20260929 03:15:20.365401 15890 tablet_bootstrap.cc:492] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd: No bootstrap required, opened a new log
I20260929 03:15:20.365482 15890 ts_tablet_manager.cc:1397] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 03:15:20.365906 15890 raft_consensus.cc:348] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:20.366015 15890 raft_consensus.cc:374] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:20.366055 15890 raft_consensus.cc:729] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63033321bdae433e9119b8e602eb9bcd, State: Initialized, Role: FOLLOWER
I20260929 03:15:20.366169 15890 consensus_queue.cc:260] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:20.366457 15890 ts_tablet_manager.cc:1428] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:20.366499 15366 catalog_manager.cc:4841] TS 63033321bdae433e9119b8e602eb9bcd (127.13.238.131:36541): tablet d7550ddcea624627bf83f1d7f323d3ff (table test-table [id=4b2067bd31a14c819a7843fb48931cac]) successfully deleted
W20260929 03:15:20.591516 15586 tablet.cc:2369] T 517c4afdc55545929c070f0bcf0a16f4 P f6d71a5c949f4e6cacb9479d6e65ee10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:15:20.766003 15906 raft_consensus.cc:482] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:15:20.766147 15906 raft_consensus.cc:504] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:20.766621 15906 leader_election.cc:288] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6d71a5c949f4e6cacb9479d6e65ee10 (127.13.238.129:33215), 88872278b28e4a6893cf11479a855e87 (127.13.238.130:35591)
I20260929 03:15:20.767150 15539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517c4afdc55545929c070f0bcf0a16f4" candidate_uuid: "63033321bdae433e9119b8e602eb9bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" is_pre_election: true
I20260929 03:15:20.767150 15679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517c4afdc55545929c070f0bcf0a16f4" candidate_uuid: "63033321bdae433e9119b8e602eb9bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88872278b28e4a6893cf11479a855e87" is_pre_election: true
I20260929 03:15:20.767309 15679 raft_consensus.cc:2454] T 517c4afdc55545929c070f0bcf0a16f4 P 88872278b28e4a6893cf11479a855e87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63033321bdae433e9119b8e602eb9bcd in term 0.
I20260929 03:15:20.767318 15539 raft_consensus.cc:2454] T 517c4afdc55545929c070f0bcf0a16f4 P f6d71a5c949f4e6cacb9479d6e65ee10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63033321bdae433e9119b8e602eb9bcd in term 0.
I20260929 03:15:20.767619 15751 leader_election.cc:302] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 63033321bdae433e9119b8e602eb9bcd, 88872278b28e4a6893cf11479a855e87, f6d71a5c949f4e6cacb9479d6e65ee10; no voters:
I20260929 03:15:20.767848 15906 raft_consensus.cc:2781] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:15:20.767920 15906 raft_consensus.cc:482] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:15:20.767967 15906 raft_consensus.cc:3037] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:20.768950 15906 raft_consensus.cc:504] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:20.769270 15906 leader_election.cc:288] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd [CANDIDATE]: Term 1 election: Requested vote from peers f6d71a5c949f4e6cacb9479d6e65ee10 (127.13.238.129:33215), 88872278b28e4a6893cf11479a855e87 (127.13.238.130:35591)
I20260929 03:15:20.769651 15679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517c4afdc55545929c070f0bcf0a16f4" candidate_uuid: "63033321bdae433e9119b8e602eb9bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88872278b28e4a6893cf11479a855e87"
I20260929 03:15:20.769634 15539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517c4afdc55545929c070f0bcf0a16f4" candidate_uuid: "63033321bdae433e9119b8e602eb9bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10"
I20260929 03:15:20.769760 15539 raft_consensus.cc:3037] T 517c4afdc55545929c070f0bcf0a16f4 P f6d71a5c949f4e6cacb9479d6e65ee10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:20.769769 15679 raft_consensus.cc:3037] T 517c4afdc55545929c070f0bcf0a16f4 P 88872278b28e4a6893cf11479a855e87 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:20.770686 15539 raft_consensus.cc:2454] T 517c4afdc55545929c070f0bcf0a16f4 P f6d71a5c949f4e6cacb9479d6e65ee10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63033321bdae433e9119b8e602eb9bcd in term 1.
I20260929 03:15:20.770725 15679 raft_consensus.cc:2454] T 517c4afdc55545929c070f0bcf0a16f4 P 88872278b28e4a6893cf11479a855e87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63033321bdae433e9119b8e602eb9bcd in term 1.
I20260929 03:15:20.770962 15750 leader_election.cc:302] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63033321bdae433e9119b8e602eb9bcd, f6d71a5c949f4e6cacb9479d6e65ee10; no voters:
I20260929 03:15:20.771126 15906 raft_consensus.cc:2781] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:20.771210 15906 raft_consensus.cc:686] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd [term 1 LEADER]: Becoming Leader. State: Replica: 63033321bdae433e9119b8e602eb9bcd, State: Running, Role: LEADER
I20260929 03:15:20.771353 15906 consensus_queue.cc:237] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:20.773057 15381 catalog_manager.cc:5495] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd reported cstate change: term changed from 0 to 1, leader changed from <none> to 63033321bdae433e9119b8e602eb9bcd (127.13.238.131). New cstate: current_term: 1 leader_uuid: "63033321bdae433e9119b8e602eb9bcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } health_report { overall_health: HEALTHY } } }
Sep 29 03:15:20 dist-test-slave-nbjg krb5kdc[15337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651719, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:20.840204 15923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:20.832056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:37979 (local address 127.0.0.1:34954)
0929 03:15:20.832351 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:20.832370 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:20.832482 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:20.832700 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:20.832702 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:20.833088 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:20.833195 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:20.833201 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.833993 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.833997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:20.834648 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:20.834658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.835099 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.835102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:20.835165 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:20.835703 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:20.835715 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:20.837201 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:20.838776 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:20.838784 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:20.838790 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:20.839028 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:20.839288 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:20.839291 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:20.839292 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:20.839376 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:20.839710 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:20.839715 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:20.839831 (+ 116us) client_negotiation.cc:769] Sending connection context
0929 03:15:20.839935 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 03:15:20.840063 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":97,"threads_started":1}
I20260929 03:15:20.840253 15924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:20.832084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34954 (local address 127.13.238.190:37979)
0929 03:15:20.832333 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:20.832342 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:20.832442 (+ 100us) server_negotiation.cc:407] Connection header received
0929 03:15:20.832555 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:20.832557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:20.832596 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:20.832669 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:20.833323 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.833872 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.834762 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:20.834997 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:20.835081 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:20.837345 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:20.837348 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:20.837350 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:20.837371 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:20.838666 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:20.839138 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:20.839141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:20.839142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:20.839215 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:20.839443 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:20.839445 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:20.839446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:20.839607 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:20.839677 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:20.839977 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 03:15:20.840131 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":180,"thread_start_us":95,"threads_started":1}
I20260929 03:15:20.844148 15381 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:34954:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 03:15:20.844844 15381 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:15:20.849820 15659 tablet_service.cc:1467] Processing CreateTablet for tablet d8fbbba17f144d53a427874b27cb02fb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e5671c80abac4b6baa826ad2dd923322]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:15:20.849828 15519 tablet_service.cc:1467] Processing CreateTablet for tablet d8fbbba17f144d53a427874b27cb02fb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e5671c80abac4b6baa826ad2dd923322]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:15:20.849831 15799 tablet_service.cc:1467] Processing CreateTablet for tablet d8fbbba17f144d53a427874b27cb02fb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e5671c80abac4b6baa826ad2dd923322]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:15:20.850267 15519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8fbbba17f144d53a427874b27cb02fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:20.850267 15799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8fbbba17f144d53a427874b27cb02fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:20.850289 15659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8fbbba17f144d53a427874b27cb02fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:20.852758 15888 tablet_bootstrap.cc:492] T d8fbbba17f144d53a427874b27cb02fb P 88872278b28e4a6893cf11479a855e87: Bootstrap starting.
I20260929 03:15:20.853176 15890 tablet_bootstrap.cc:492] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd: Bootstrap starting.
I20260929 03:15:20.853619 15888 tablet_bootstrap.cc:654] T d8fbbba17f144d53a427874b27cb02fb P 88872278b28e4a6893cf11479a855e87: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:20.854032 15890 tablet_bootstrap.cc:654] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:20.854133 15889 tablet_bootstrap.cc:492] T d8fbbba17f144d53a427874b27cb02fb P f6d71a5c949f4e6cacb9479d6e65ee10: Bootstrap starting.
I20260929 03:15:20.854831 15888 tablet_bootstrap.cc:492] T d8fbbba17f144d53a427874b27cb02fb P 88872278b28e4a6893cf11479a855e87: No bootstrap required, opened a new log
I20260929 03:15:20.854911 15888 ts_tablet_manager.cc:1397] T d8fbbba17f144d53a427874b27cb02fb P 88872278b28e4a6893cf11479a855e87: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:15:20.854981 15889 tablet_bootstrap.cc:654] T d8fbbba17f144d53a427874b27cb02fb P f6d71a5c949f4e6cacb9479d6e65ee10: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:20.855149 15890 tablet_bootstrap.cc:492] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd: No bootstrap required, opened a new log
I20260929 03:15:20.855208 15890 ts_tablet_manager.cc:1397] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:15:20.855362 15888 raft_consensus.cc:348] T d8fbbba17f144d53a427874b27cb02fb P 88872278b28e4a6893cf11479a855e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:20.855471 15888 raft_consensus.cc:374] T d8fbbba17f144d53a427874b27cb02fb P 88872278b28e4a6893cf11479a855e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:20.855504 15888 raft_consensus.cc:729] T d8fbbba17f144d53a427874b27cb02fb P 88872278b28e4a6893cf11479a855e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88872278b28e4a6893cf11479a855e87, State: Initialized, Role: FOLLOWER
I20260929 03:15:20.855585 15890 raft_consensus.cc:348] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:20.855679 15890 raft_consensus.cc:374] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:20.855633 15888 consensus_queue.cc:260] T d8fbbba17f144d53a427874b27cb02fb P 88872278b28e4a6893cf11479a855e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:20.855700 15890 raft_consensus.cc:729] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63033321bdae433e9119b8e602eb9bcd, State: Initialized, Role: FOLLOWER
I20260929 03:15:20.855808 15890 consensus_queue.cc:260] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:20.855886 15888 ts_tablet_manager.cc:1428] T d8fbbba17f144d53a427874b27cb02fb P 88872278b28e4a6893cf11479a855e87: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:20.856057 15890 ts_tablet_manager.cc:1428] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:20.856040 15906 tablet_replica.cc:439] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } } }
I20260929 03:15:20.856076 15896 tablet_replica.cc:439] T d8fbbba17f144d53a427874b27cb02fb P 88872278b28e4a6893cf11479a855e87: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } } }
I20260929 03:15:20.856197 15906 tablet_replica.cc:442] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:15:20.856204 15896 tablet_replica.cc:442] T d8fbbba17f144d53a427874b27cb02fb P 88872278b28e4a6893cf11479a855e87: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:15:20.856868 15889 tablet_bootstrap.cc:492] T d8fbbba17f144d53a427874b27cb02fb P f6d71a5c949f4e6cacb9479d6e65ee10: No bootstrap required, opened a new log
I20260929 03:15:20.856940 15889 ts_tablet_manager.cc:1397] T d8fbbba17f144d53a427874b27cb02fb P f6d71a5c949f4e6cacb9479d6e65ee10: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 03:15:20.857251 15889 raft_consensus.cc:348] T d8fbbba17f144d53a427874b27cb02fb P f6d71a5c949f4e6cacb9479d6e65ee10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:20.857329 15889 raft_consensus.cc:374] T d8fbbba17f144d53a427874b27cb02fb P f6d71a5c949f4e6cacb9479d6e65ee10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:20.857348 15889 raft_consensus.cc:729] T d8fbbba17f144d53a427874b27cb02fb P f6d71a5c949f4e6cacb9479d6e65ee10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d71a5c949f4e6cacb9479d6e65ee10, State: Initialized, Role: FOLLOWER
I20260929 03:15:20.857443 15889 consensus_queue.cc:260] T d8fbbba17f144d53a427874b27cb02fb P f6d71a5c949f4e6cacb9479d6e65ee10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:20.857687 15889 ts_tablet_manager.cc:1428] T d8fbbba17f144d53a427874b27cb02fb P f6d71a5c949f4e6cacb9479d6e65ee10: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 03:15:20.857658 15894 tablet_replica.cc:439] T d8fbbba17f144d53a427874b27cb02fb P f6d71a5c949f4e6cacb9479d6e65ee10: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } } }
I20260929 03:15:20.857810 15894 tablet_replica.cc:442] T d8fbbba17f144d53a427874b27cb02fb P f6d71a5c949f4e6cacb9479d6e65ee10: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:15:21.094010 15906 raft_consensus.cc:482] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:15:21.094147 15906 raft_consensus.cc:504] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:21.094621 15906 leader_election.cc:288] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6d71a5c949f4e6cacb9479d6e65ee10 (127.13.238.129:33215), 88872278b28e4a6893cf11479a855e87 (127.13.238.130:35591)
I20260929 03:15:21.095142 15539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fbbba17f144d53a427874b27cb02fb" candidate_uuid: "63033321bdae433e9119b8e602eb9bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" is_pre_election: true
I20260929 03:15:21.095136 15679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fbbba17f144d53a427874b27cb02fb" candidate_uuid: "63033321bdae433e9119b8e602eb9bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88872278b28e4a6893cf11479a855e87" is_pre_election: true
I20260929 03:15:21.095279 15539 raft_consensus.cc:2454] T d8fbbba17f144d53a427874b27cb02fb P f6d71a5c949f4e6cacb9479d6e65ee10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63033321bdae433e9119b8e602eb9bcd in term 0.
I20260929 03:15:21.095289 15679 raft_consensus.cc:2454] T d8fbbba17f144d53a427874b27cb02fb P 88872278b28e4a6893cf11479a855e87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63033321bdae433e9119b8e602eb9bcd in term 0.
I20260929 03:15:21.095563 15750 leader_election.cc:302] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63033321bdae433e9119b8e602eb9bcd, f6d71a5c949f4e6cacb9479d6e65ee10; no voters:
I20260929 03:15:21.095758 15906 raft_consensus.cc:2781] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:15:21.095820 15906 raft_consensus.cc:482] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:15:21.095858 15906 raft_consensus.cc:3037] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:21.096817 15906 raft_consensus.cc:504] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:21.097162 15906 leader_election.cc:288] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd [CANDIDATE]: Term 1 election: Requested vote from peers f6d71a5c949f4e6cacb9479d6e65ee10 (127.13.238.129:33215), 88872278b28e4a6893cf11479a855e87 (127.13.238.130:35591)
I20260929 03:15:21.097559 15539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fbbba17f144d53a427874b27cb02fb" candidate_uuid: "63033321bdae433e9119b8e602eb9bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10"
I20260929 03:15:21.097559 15679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fbbba17f144d53a427874b27cb02fb" candidate_uuid: "63033321bdae433e9119b8e602eb9bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88872278b28e4a6893cf11479a855e87"
I20260929 03:15:21.097697 15679 raft_consensus.cc:3037] T d8fbbba17f144d53a427874b27cb02fb P 88872278b28e4a6893cf11479a855e87 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:21.097697 15539 raft_consensus.cc:3037] T d8fbbba17f144d53a427874b27cb02fb P f6d71a5c949f4e6cacb9479d6e65ee10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:21.098883 15539 raft_consensus.cc:2454] T d8fbbba17f144d53a427874b27cb02fb P f6d71a5c949f4e6cacb9479d6e65ee10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63033321bdae433e9119b8e602eb9bcd in term 1.
I20260929 03:15:21.098886 15679 raft_consensus.cc:2454] T d8fbbba17f144d53a427874b27cb02fb P 88872278b28e4a6893cf11479a855e87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63033321bdae433e9119b8e602eb9bcd in term 1.
I20260929 03:15:21.099180 15751 leader_election.cc:302] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63033321bdae433e9119b8e602eb9bcd, 88872278b28e4a6893cf11479a855e87; no voters:
I20260929 03:15:21.099356 15906 raft_consensus.cc:2781] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:21.099450 15906 raft_consensus.cc:686] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd [term 1 LEADER]: Becoming Leader. State: Replica: 63033321bdae433e9119b8e602eb9bcd, State: Running, Role: LEADER
I20260929 03:15:21.099598 15906 consensus_queue.cc:237] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } }
I20260929 03:15:21.100242 15898 tablet_replica.cc:439] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd: TxnStatusTablet state changed. Reason: New leader 63033321bdae433e9119b8e602eb9bcd. Latest consensus state: current_term: 1 leader_uuid: "63033321bdae433e9119b8e602eb9bcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } } }
I20260929 03:15:21.100334 15898 tablet_replica.cc:442] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:15:21.100560 15929 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 03:15:21.100762 15929 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 03:15:21.101423 15381 catalog_manager.cc:5495] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd reported cstate change: term changed from 0 to 1, leader changed from <none> to 63033321bdae433e9119b8e602eb9bcd (127.13.238.131). New cstate: current_term: 1 leader_uuid: "63033321bdae433e9119b8e602eb9bcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } health_report { overall_health: HEALTHY } } }
I20260929 03:15:21.154117 15923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:21.150646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:36541 (local address 127.0.0.1:32800)
0929 03:15:21.151091 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:21.151106 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:21.151226 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:21.151547 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:21.151550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:21.151559 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:15:21.151665 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:21.151669 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:21.152376 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:21.152379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:21.152990 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:21.152998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:21.153427 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:21.153431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:21.153518 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:21.153565 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:15:21.153834 (+ 269us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:15:21.153848 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 03:15:21.153901 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 03:15:21.153994 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316}
I20260929 03:15:21.154122 15932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:21.150785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32800 (local address 127.13.238.131:36541)
0929 03:15:21.151027 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:21.151032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:21.151165 (+ 133us) server_negotiation.cc:407] Connection header received
0929 03:15:21.151354 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:21.151358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:21.151414 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:21.151501 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:21.151758 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:21.152279 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:21.153097 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:21.153328 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:21.153416 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:21.153686 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:21.153762 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:21.153818 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:21.153920 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 03:15:21.153971 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":80,"threads_started":1}
I20260929 03:15:21.156653 15679 raft_consensus.cc:1261] T d8fbbba17f144d53a427874b27cb02fb P 88872278b28e4a6893cf11479a855e87 [term 1 FOLLOWER]: Refusing update from remote peer 63033321bdae433e9119b8e602eb9bcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:21.156729 15539 raft_consensus.cc:1261] T d8fbbba17f144d53a427874b27cb02fb P f6d71a5c949f4e6cacb9479d6e65ee10 [term 1 FOLLOWER]: Refusing update from remote peer 63033321bdae433e9119b8e602eb9bcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:21.157243 15906 consensus_queue.cc:1035] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:21.157411 15898 consensus_queue.cc:1035] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:21.158959 15894 tablet_replica.cc:439] T d8fbbba17f144d53a427874b27cb02fb P f6d71a5c949f4e6cacb9479d6e65ee10: TxnStatusTablet state changed. Reason: New leader 63033321bdae433e9119b8e602eb9bcd. Latest consensus state: current_term: 1 leader_uuid: "63033321bdae433e9119b8e602eb9bcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } } }
I20260929 03:15:21.159080 15894 tablet_replica.cc:442] T d8fbbba17f144d53a427874b27cb02fb P f6d71a5c949f4e6cacb9479d6e65ee10: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:15:21.159606 15896 tablet_replica.cc:439] T d8fbbba17f144d53a427874b27cb02fb P 88872278b28e4a6893cf11479a855e87: TxnStatusTablet state changed. Reason: New leader 63033321bdae433e9119b8e602eb9bcd. Latest consensus state: current_term: 1 leader_uuid: "63033321bdae433e9119b8e602eb9bcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } } }
I20260929 03:15:21.159725 15896 tablet_replica.cc:442] T d8fbbba17f144d53a427874b27cb02fb P 88872278b28e4a6893cf11479a855e87: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:15:21.162065 15898 tablet_replica.cc:439] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63033321bdae433e9119b8e602eb9bcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } } }
I20260929 03:15:21.162179 15898 tablet_replica.cc:442] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:15:21.162313 15898 tablet_replica.cc:439] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63033321bdae433e9119b8e602eb9bcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } } }
I20260929 03:15:21.162381 15898 tablet_replica.cc:442] T d8fbbba17f144d53a427874b27cb02fb P 63033321bdae433e9119b8e602eb9bcd: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:15:21.163130 15896 tablet_replica.cc:439] T d8fbbba17f144d53a427874b27cb02fb P 88872278b28e4a6893cf11479a855e87: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "63033321bdae433e9119b8e602eb9bcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } } }
I20260929 03:15:21.163237 15896 tablet_replica.cc:442] T d8fbbba17f144d53a427874b27cb02fb P 88872278b28e4a6893cf11479a855e87: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:15:21.165540 15894 tablet_replica.cc:439] T d8fbbba17f144d53a427874b27cb02fb P f6d71a5c949f4e6cacb9479d6e65ee10: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "63033321bdae433e9119b8e602eb9bcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 } } peers { permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 } } peers { permanent_uuid: "63033321bdae433e9119b8e602eb9bcd" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36541 } } }
I20260929 03:15:21.165652 15894 tablet_replica.cc:442] T d8fbbba17f144d53a427874b27cb02fb P f6d71a5c949f4e6cacb9479d6e65ee10: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:15:21.166574 15935 mvcc.cc:205] Tried to move back new op lower bound from 7334509449853706240 to 7334509449625333760. Current Snapshot: MvccSnapshot[applied={T|T < 7334509449853706240 or (T in {7334509449853706240})}]
I20260929 03:15:21.178313 15945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:21.174955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:36541 (local address 127.13.238.131:57977)
0929 03:15:21.175194 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:21.175207 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:21.175303 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:21.175621 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:21.175624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:21.175633 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:15:21.175733 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:21.175738 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:21.176530 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:21.176533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:21.177172 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:21.177181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:21.177593 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:21.177596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:21.177698 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:21.177741 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:15:21.178004 (+ 263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:15:21.178010 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:15:21.178067 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 03:15:21.178141 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":25,"thread_start_us":83,"threads_started":1}
I20260929 03:15:21.178313 15932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:21.175022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:57977 (local address 127.13.238.131:36541)
0929 03:15:21.175155 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:21.175160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:21.175294 (+ 134us) server_negotiation.cc:407] Connection header received
0929 03:15:21.175401 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:21.175404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:21.175526 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:21.175603 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:21.175847 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:21.176433 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:21.177282 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:21.177504 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:21.177585 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:21.177869 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:21.177936 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:21.177990 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:21.178073 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 03:15:21.178122 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":55}
I20260929 03:15:21.185294 15539 raft_consensus.cc:1261] T 517c4afdc55545929c070f0bcf0a16f4 P f6d71a5c949f4e6cacb9479d6e65ee10 [term 1 FOLLOWER]: Refusing update from remote peer 63033321bdae433e9119b8e602eb9bcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:21.185294 15679 raft_consensus.cc:1261] T 517c4afdc55545929c070f0bcf0a16f4 P 88872278b28e4a6893cf11479a855e87 [term 1 FOLLOWER]: Refusing update from remote peer 63033321bdae433e9119b8e602eb9bcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:21.185671 15940 consensus_queue.cc:1035] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d71a5c949f4e6cacb9479d6e65ee10" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 33215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:21.185844 15898 consensus_queue.cc:1035] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "88872278b28e4a6893cf11479a855e87" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:21.189337 15934 mvcc.cc:205] Tried to move back new op lower bound from 7334509449971732480 to 7334509448280690688. Current Snapshot: MvccSnapshot[applied={T|T < 7334509449971732480}]
I20260929 03:15:21.231829 15381 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:21.231999 15381 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:21.234042 15381 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1324724fd0e94195be72e9cedb9c27ef: Sending DeleteTablet for 3 replicas of tablet 517c4afdc55545929c070f0bcf0a16f4
I20260929 03:15:21.234671 15799 tablet_service.cc:1514] Processing DeleteTablet for tablet 517c4afdc55545929c070f0bcf0a16f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:21 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:32778
I20260929 03:15:21.234679 15659 tablet_service.cc:1514] Processing DeleteTablet for tablet 517c4afdc55545929c070f0bcf0a16f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:21 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:56284
I20260929 03:15:21.234730 15519 tablet_service.cc:1514] Processing DeleteTablet for tablet 517c4afdc55545929c070f0bcf0a16f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:21 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:40030
I20260929 03:15:21.235114 15953 tablet_replica.cc:330] T 517c4afdc55545929c070f0bcf0a16f4 P f6d71a5c949f4e6cacb9479d6e65ee10: stopping tablet replica
I20260929 03:15:21.235173 15951 tablet_replica.cc:330] T 517c4afdc55545929c070f0bcf0a16f4 P 88872278b28e4a6893cf11479a855e87: stopping tablet replica
I20260929 03:15:21.235297 15953 raft_consensus.cc:2229] T 517c4afdc55545929c070f0bcf0a16f4 P f6d71a5c949f4e6cacb9479d6e65ee10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:15:21.235322 15951 raft_consensus.cc:2229] T 517c4afdc55545929c070f0bcf0a16f4 P 88872278b28e4a6893cf11479a855e87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:15:21.235481 15953 raft_consensus.cc:2258] T 517c4afdc55545929c070f0bcf0a16f4 P f6d71a5c949f4e6cacb9479d6e65ee10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:21.235481 15951 raft_consensus.cc:2258] T 517c4afdc55545929c070f0bcf0a16f4 P 88872278b28e4a6893cf11479a855e87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:21.236143 15951 ts_tablet_manager.cc:1905] T 517c4afdc55545929c070f0bcf0a16f4 P 88872278b28e4a6893cf11479a855e87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:21.236804 15953 ts_tablet_manager.cc:1905] T 517c4afdc55545929c070f0bcf0a16f4 P f6d71a5c949f4e6cacb9479d6e65ee10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:21.239271 15951 ts_tablet_manager.cc:1918] T 517c4afdc55545929c070f0bcf0a16f4 P 88872278b28e4a6893cf11479a855e87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:15:21.239359 15951 log.cc:1196] T 517c4afdc55545929c070f0bcf0a16f4 P 88872278b28e4a6893cf11479a855e87: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-1/wal/wals/517c4afdc55545929c070f0bcf0a16f4
I20260929 03:15:21.239575 15952 tablet_replica.cc:330] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd: stopping tablet replica
I20260929 03:15:21.239785 15952 raft_consensus.cc:2229] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:15:21.239827 15953 ts_tablet_manager.cc:1918] T 517c4afdc55545929c070f0bcf0a16f4 P f6d71a5c949f4e6cacb9479d6e65ee10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:15:21.239926 15953 log.cc:1196] T 517c4afdc55545929c070f0bcf0a16f4 P f6d71a5c949f4e6cacb9479d6e65ee10: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-0/wal/wals/517c4afdc55545929c070f0bcf0a16f4
I20260929 03:15:21.240125 15952 raft_consensus.cc:2258] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:21.240222 15953 ts_tablet_manager.cc:1939] T 517c4afdc55545929c070f0bcf0a16f4 P f6d71a5c949f4e6cacb9479d6e65ee10: Deleting consensus metadata
I20260929 03:15:21.240867 15952 ts_tablet_manager.cc:1905] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:21.241008 15366 catalog_manager.cc:4841] TS f6d71a5c949f4e6cacb9479d6e65ee10 (127.13.238.129:33215): tablet 517c4afdc55545929c070f0bcf0a16f4 (table test-table [id=bdc00811df0b4165aadae2777c2deeb1]) successfully deleted
I20260929 03:15:21.243153 15952 ts_tablet_manager.cc:1918] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:15:21.243299 15952 log.cc:1196] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/517c4afdc55545929c070f0bcf0a16f4
I20260929 03:15:21.243515 15951 ts_tablet_manager.cc:1939] T 517c4afdc55545929c070f0bcf0a16f4 P 88872278b28e4a6893cf11479a855e87: Deleting consensus metadata
I20260929 03:15:21.243621 15952 ts_tablet_manager.cc:1939] T 517c4afdc55545929c070f0bcf0a16f4 P 63033321bdae433e9119b8e602eb9bcd: Deleting consensus metadata
I20260929 03:15:21.244331 15367 catalog_manager.cc:4841] TS 88872278b28e4a6893cf11479a855e87 (127.13.238.130:35591): tablet 517c4afdc55545929c070f0bcf0a16f4 (table test-table [id=bdc00811df0b4165aadae2777c2deeb1]) successfully deleted
I20260929 03:15:21.244400 15366 catalog_manager.cc:4841] TS 63033321bdae433e9119b8e602eb9bcd (127.13.238.131:36541): tablet 517c4afdc55545929c070f0bcf0a16f4 (table test-table [id=bdc00811df0b4165aadae2777c2deeb1]) successfully deleted
Sep 29 03:15:21 dist-test-slave-nbjg krb5kdc[15337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651720, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.238.129@KRBTEST.COM
I20260929 03:15:21.250340 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:21.236726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40034 (local address 127.13.238.129:33215)
0929 03:15:21.239627 (+ 2901us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:21.239632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:21.239877 (+ 245us) server_negotiation.cc:407] Connection header received
0929 03:15:21.240055 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:21.240058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:21.240106 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:21.240177 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:21.241101 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:21.241958 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:21.243226 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:21.243567 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:21.243673 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:21.246793 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:21.246804 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:21.246819 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:21.246843 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:21.248379 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:21.248884 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:21.248890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:21.248891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:21.248971 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:21.249258 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:21.249260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:21.249262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:21.249517 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:21.249722 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:21.249994 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 03:15:21.250111 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2802,"thread_start_us":67,"threads_started":1}
W20260929 03:15:21.250771 15579 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40034
I20260929 03:15:21.259101 15924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:21.252391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34968 (local address 127.13.238.190:37979)
0929 03:15:21.252544 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:21.252547 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:21.252691 (+ 144us) server_negotiation.cc:407] Connection header received
0929 03:15:21.252789 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:21.252791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:21.252831 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:21.252900 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:21.253467 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:21.253992 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:21.254798 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:21.255044 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:21.255127 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:21.256330 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:21.256333 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:21.256335 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:21.256361 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:21.257578 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:21.257971 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:21.257974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:21.257975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:21.258049 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:21.258315 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:21.258319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:21.258320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:21.258516 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:21.258629 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:21.258830 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 03:15:21.258934 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 03:15:21.259508 15381 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34968
I20260929 03:15:21.260504 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 15454
I20260929 03:15:21.266692 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 15594
I20260929 03:15:21.272043 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 15734
I20260929 03:15:21.278091 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 15350
2026-09-29T03:15:21Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3223 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:15:21 dist-test-slave-nbjg krb5kdc[15961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:15:21 dist-test-slave-nbjg krb5kdc[15961](info): set up 2 sockets
Sep 29 03:15:21 dist-test-slave-nbjg krb5kdc[15961](info): commencing operation
krb5kdc: starting...
W20260929 03:15:22.226028 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:15:22 dist-test-slave-nbjg krb5kdc[15961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:15:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:15:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:15:22.340864 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:45251
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:45251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:22.453289 15974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:22.453565 15974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:22.453620 15974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:22.457607 15974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:22.457664 15974 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 03:15:22.457687 15974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:22.457710 15974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:22.457731 15974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:22.457787 15974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:15:22.463025 15974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:45251
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:45251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:22.464075 15974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:22.465133 15974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:22.473711 15979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:22.473717 15980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:22.473726 15982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:22.473928 15974 server_base.cc:1034] running on GCE node
I20260929 03:15:22.474664 15974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:22.475835 15974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:22.477003 15974 hybrid_clock.cc:648] HybridClock initialized: now 1790651722476985 us; error 34 us; skew 500 ppm
Sep 29 03:15:22 dist-test-slave-nbjg krb5kdc[15961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651722, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:22.479745 15974 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:15:22.480679 15974 webserver.cc:460] Webserver started at http://127.13.238.190:34419/ using document root <none> and password file <none>
I20260929 03:15:22.481280 15974 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:22.481348 15974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:22.481556 15974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:22.483701 15974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "c14f64fa6dcb41e482ec0c1c203b41bc"
format_stamp: "Formatted at 2026-09-29 03:15:22 on dist-test-slave-nbjg"
server_key: "caad27d754f7b0d909fb0cb858883659"
server_key_iv: "9ec39b6bec95be675ea0c7d469895a17"
server_key_version: "encryptionkey@0"
I20260929 03:15:22.484339 15974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "c14f64fa6dcb41e482ec0c1c203b41bc"
format_stamp: "Formatted at 2026-09-29 03:15:22 on dist-test-slave-nbjg"
server_key: "caad27d754f7b0d909fb0cb858883659"
server_key_iv: "9ec39b6bec95be675ea0c7d469895a17"
server_key_version: "encryptionkey@0"
I20260929 03:15:22.488253 15974 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:15:22.490799 15989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:22.492048 15974 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:22.492167 15974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "c14f64fa6dcb41e482ec0c1c203b41bc"
format_stamp: "Formatted at 2026-09-29 03:15:22 on dist-test-slave-nbjg"
server_key: "caad27d754f7b0d909fb0cb858883659"
server_key_iv: "9ec39b6bec95be675ea0c7d469895a17"
server_key_version: "encryptionkey@0"
I20260929 03:15:22.492269 15974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:22.514185 15974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:22.517393 15974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:22.517583 15974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:22.525451 15974 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:45251
I20260929 03:15:22.525468 16051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:45251 every 8 connection(s)
I20260929 03:15:22.526522 15974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:15:22.529341 16052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:22.535175 16052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc: Bootstrap starting.
I20260929 03:15:22.536676 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 15974
I20260929 03:15:22.536842 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:15:22.537097 14266 external_mini_cluster.cc:1428] Setting key e0870dfd7edd9af323d1269272a21c73
I20260929 03:15:22.537904 16052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:22.538832 16052 log.cc:824] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:22.540937 16052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc: No bootstrap required, opened a new log
I20260929 03:15:22.545024 16052 raft_consensus.cc:348] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c14f64fa6dcb41e482ec0c1c203b41bc" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 45251 } }
I20260929 03:15:22.545324 16052 raft_consensus.cc:374] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:22.545379 16052 raft_consensus.cc:729] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c14f64fa6dcb41e482ec0c1c203b41bc, State: Initialized, Role: FOLLOWER
I20260929 03:15:22.545854 16052 consensus_queue.cc:260] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c14f64fa6dcb41e482ec0c1c203b41bc" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 45251 } }
I20260929 03:15:22.545989 16052 raft_consensus.cc:388] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:15:22.546047 16052 raft_consensus.cc:482] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:15:22.546141 16052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc [term 0 FOLLOWER]: Advancing to term 1
Sep 29 03:15:22 dist-test-slave-nbjg krb5kdc[15961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:22.547191 16052 raft_consensus.cc:504] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c14f64fa6dcb41e482ec0c1c203b41bc" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 45251 } }
I20260929 03:15:22.547573 16052 leader_election.cc:302] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c14f64fa6dcb41e482ec0c1c203b41bc; no voters:
I20260929 03:15:22.547919 16052 leader_election.cc:288] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:15:22.548024 16057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:22.548235 16057 raft_consensus.cc:686] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc [term 1 LEADER]: Becoming Leader. State: Replica: c14f64fa6dcb41e482ec0c1c203b41bc, State: Running, Role: LEADER
I20260929 03:15:22.548619 16057 consensus_queue.cc:237] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c14f64fa6dcb41e482ec0c1c203b41bc" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 45251 } }
I20260929 03:15:22.549345 16052 sys_catalog.cc:564] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:22.550344 16058 sys_catalog.cc:455] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c14f64fa6dcb41e482ec0c1c203b41bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c14f64fa6dcb41e482ec0c1c203b41bc" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 45251 } } }
I20260929 03:15:22.550491 16058 sys_catalog.cc:458] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:22.550849 16059 sys_catalog.cc:455] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c14f64fa6dcb41e482ec0c1c203b41bc. Latest consensus state: current_term: 1 leader_uuid: "c14f64fa6dcb41e482ec0c1c203b41bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c14f64fa6dcb41e482ec0c1c203b41bc" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 45251 } } }
I20260929 03:15:22.551016 16059 sys_catalog.cc:458] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:22.552858 16055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:22.538253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52970 (local address 127.13.238.190:45251)
0929 03:15:22.538675 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:22.538691 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:22.538727 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:15:22.539418 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:22.539437 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:22.539770 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:22.540054 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:22.540895 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:22.541971 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:22.543236 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:22.543849 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:22.543978 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:22.546772 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:22.546779 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:22.546797 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:22.546831 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:22.548924 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:22.549476 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:22.549483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:22.549489 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:22.549627 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:22.549965 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:22.549969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:22.549970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:22.550551 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:22.551420 (+ 869us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:22.551822 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 03:15:22.552107 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":125,"threads_started":1}
W20260929 03:15:22.554518 16072 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:15:22.554594 16072 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:15:22.554649 16066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:15:22.555472 16066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:15:22.561278 16066 catalog_manager.cc:1347] Generated new cluster ID: 73cd6d65b2574a7e8b5ac5771dd48f69
I20260929 03:15:22.561381 16066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:15:22.568789 16066 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:15:22.569732 16066 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:15:22.579442 16066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc: Generated new TSK 0
I20260929 03:15:22.580180 16066 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
I20260929 03:15:22.609164 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:0
--local_ip_for_outbound_sockets=127.13.238.129
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:45251
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:22.719770 16078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:22.720041 16078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:22.720104 16078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:22.724148 16078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:22.724202 16078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:22.724251 16078 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 03:15:22.724277 16078 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:15:22.724330 16078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:15:22.729466 16078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.238.190:45251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:22.730530 16078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:22.731568 16078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:22.740979 16086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:22.740945 16084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:22.740958 16078 server_base.cc:1034] running on GCE node
W20260929 03:15:22.740985 16083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:22.741518 16078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:22.742151 16078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:22.743336 16078 hybrid_clock.cc:648] HybridClock initialized: now 1790651722743316 us; error 28 us; skew 500 ppm
Sep 29 03:15:22 dist-test-slave-nbjg krb5kdc[15961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651722, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:22.746031 16078 init.cc:377] Logged in from keytab as kudu/127.13.238.129@KRBTEST.COM (short username kudu)
I20260929 03:15:22.746927 16078 webserver.cc:460] Webserver started at http://127.13.238.129:34765/ using document root <none> and password file <none>
I20260929 03:15:22.747448 16078 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:22.747503 16078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:22.747673 16078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:22.749711 16078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-0/data/instance:
uuid: "10c9b446b29f4e8f966a0c34278bc522"
format_stamp: "Formatted at 2026-09-29 03:15:22 on dist-test-slave-nbjg"
server_key: "ecdebda30e9b88d196e8a7bd3b2eee64"
server_key_iv: "c9172138cdf0f3965e7e20525e3bb09d"
server_key_version: "encryptionkey@0"
I20260929 03:15:22.750319 16078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance:
uuid: "10c9b446b29f4e8f966a0c34278bc522"
format_stamp: "Formatted at 2026-09-29 03:15:22 on dist-test-slave-nbjg"
server_key: "ecdebda30e9b88d196e8a7bd3b2eee64"
server_key_iv: "c9172138cdf0f3965e7e20525e3bb09d"
server_key_version: "encryptionkey@0"
I20260929 03:15:22.754010 16078 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:15:22.756424 16093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:22.757591 16078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:15:22.757690 16078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "10c9b446b29f4e8f966a0c34278bc522"
format_stamp: "Formatted at 2026-09-29 03:15:22 on dist-test-slave-nbjg"
server_key: "ecdebda30e9b88d196e8a7bd3b2eee64"
server_key_iv: "c9172138cdf0f3965e7e20525e3bb09d"
server_key_version: "encryptionkey@0"
I20260929 03:15:22.757810 16078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:22.772241 16078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:22.775544 16078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:22.775733 16078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:22.777307 16078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:22.777361 16078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:22.777405 16078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:22.777419 16078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:15:22 dist-test-slave-nbjg krb5kdc[15961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651722, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:22.787873 16078 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:34855
I20260929 03:15:22.787935 16208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:34855 every 8 connection(s)
I20260929 03:15:22.789008 16078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:15:22.791366 16055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:22.778087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:50015 (local address 127.13.238.190:45251)
0929 03:15:22.778289 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:22.778294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:22.779256 (+ 962us) server_negotiation.cc:407] Connection header received
0929 03:15:22.780533 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:22.780537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:22.780602 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:22.780710 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:22.782171 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:22.782822 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:22.784094 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:22.784359 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:22.784458 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:22.787467 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:22.787470 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:22.787473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:22.787501 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:22.789278 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:22.789922 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:22.789926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:22.789928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:22.790019 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:22.790382 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:22.790384 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:22.790386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:22.790570 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:22.790644 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:22.791121 (+ 477us) server_negotiation.cc:299] Negotiation successful
0929 03:15:22.791239 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 03:15:22.792207 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:22.778536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:45251 (local address 127.13.238.129:50015)
0929 03:15:22.779057 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:22.779121 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:22.780246 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:22.780986 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:22.780994 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:22.781350 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:22.782005 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:22.782023 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:22.782947 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:22.782950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:22.783973 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:22.783984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:22.784560 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:22.784565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:22.784729 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:22.785365 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:22.785382 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:22.787256 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:22.789430 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:22.789437 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:22.789453 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:22.789803 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:22.790114 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:22.790117 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:22.790119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:22.790305 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:22.790649 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:22.790654 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:22.790893 (+ 239us) client_negotiation.cc:769] Sending connection context
0929 03:15:22.791045 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 03:15:22.791301 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":103,"threads_started":1}
I20260929 03:15:22.793682 16209 heartbeater.cc:347] Connected to a master server at 127.13.238.190:45251
I20260929 03:15:22.793829 16209 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:22.794371 16209 heartbeater.cc:511] Master 127.13.238.190:45251 requested a full tablet report, sending...
I20260929 03:15:22.795272 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 16078
I20260929 03:15:22.795394 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance
I20260929 03:15:22.795645 14266 external_mini_cluster.cc:1428] Setting key c6f4978924b1a2fbbcc28d971104c44e
I20260929 03:15:22.796870 16006 ts_manager.cc:194] Registered new tserver with Master: 10c9b446b29f4e8f966a0c34278bc522 (127.13.238.129:34855)
I20260929 03:15:22.798224 16006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.129@KRBTEST.COM'} at 127.13.238.129:50015
I20260929 03:15:22.803293 16055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:22.795811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:34685 (local address 127.13.238.190:45251)
0929 03:15:22.796008 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:22.796012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:22.796076 (+ 64us) server_negotiation.cc:407] Connection header received
0929 03:15:22.796210 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:22.796213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:22.796257 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:22.796337 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:22.796823 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:22.797575 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:22.798750 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:22.798990 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:22.799077 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:22.800201 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:22.800205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:22.800207 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:22.800230 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:22.801694 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:22.802094 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:22.802097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:22.802099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:22.802181 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:22.802493 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:22.802496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:22.802497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:22.802666 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:22.802753 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:22.803046 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 03:15:22.803151 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:15:22.803301 16215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:22.795747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:45251 (local address 127.13.238.129:34685)
0929 03:15:22.796008 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:22.796024 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:22.796118 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:22.796356 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:22.796358 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:22.796583 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:22.796699 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:22.796706 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:22.797723 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:22.797727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:22.798602 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:22.798615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:22.799099 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:22.799103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:22.799251 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:22.799716 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:22.799731 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:22.800061 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:22.801811 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:22.801814 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:22.801815 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:22.802009 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:22.802304 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:22.802307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:22.802309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:22.802399 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:22.802771 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:22.802774 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:22.802854 (+ 80us) client_negotiation.cc:769] Sending connection context
0929 03:15:22.802972 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 03:15:22.803108 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.13.238.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
I20260929 03:15:22.825163 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:0
--local_ip_for_outbound_sockets=127.13.238.130
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:45251
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:22.935842 16218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:22.936091 16218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:22.936137 16218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:22.940044 16218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:22.940088 16218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:22.940124 16218 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 03:15:22.940140 16218 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:15:22.940173 16218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:15:22.945201 16218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.238.190:45251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:22.946128 16218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:22.947144 16218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:15:22.956709 16218 server_base.cc:1034] running on GCE node
W20260929 03:15:22.956626 16226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:22.956648 16224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:22.956607 16223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:22.957231 16218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:22.957921 16218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:22.959082 16218 hybrid_clock.cc:648] HybridClock initialized: now 1790651722959044 us; error 51 us; skew 500 ppm
Sep 29 03:15:22 dist-test-slave-nbjg krb5kdc[15961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651722, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:22.961860 16218 init.cc:377] Logged in from keytab as kudu/127.13.238.130@KRBTEST.COM (short username kudu)
I20260929 03:15:22.962739 16218 webserver.cc:460] Webserver started at http://127.13.238.130:37847/ using document root <none> and password file <none>
I20260929 03:15:22.963274 16218 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:22.963330 16218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:22.963505 16218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:22.965509 16218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-1/data/instance:
uuid: "b57d5c8da10b4c4ca970f4342273a6df"
format_stamp: "Formatted at 2026-09-29 03:15:22 on dist-test-slave-nbjg"
server_key: "f4012b273361b51fb302587c45e8452b"
server_key_iv: "8958d3cbd8bcece860135ba2d60bc736"
server_key_version: "encryptionkey@0"
I20260929 03:15:22.966019 16218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance:
uuid: "b57d5c8da10b4c4ca970f4342273a6df"
format_stamp: "Formatted at 2026-09-29 03:15:22 on dist-test-slave-nbjg"
server_key: "f4012b273361b51fb302587c45e8452b"
server_key_iv: "8958d3cbd8bcece860135ba2d60bc736"
server_key_version: "encryptionkey@0"
I20260929 03:15:22.969681 16218 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:15:22.972072 16233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:22.973249 16218 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:15:22.973345 16218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "b57d5c8da10b4c4ca970f4342273a6df"
format_stamp: "Formatted at 2026-09-29 03:15:22 on dist-test-slave-nbjg"
server_key: "f4012b273361b51fb302587c45e8452b"
server_key_iv: "8958d3cbd8bcece860135ba2d60bc736"
server_key_version: "encryptionkey@0"
I20260929 03:15:22.973430 16218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:22.988492 16218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:22.991776 16218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:22.991928 16218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:22.993382 16218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:22.993438 16218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:22.993491 16218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:22.993534 16218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:15:22 dist-test-slave-nbjg krb5kdc[15961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651722, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:23.004041 16218 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:37509
I20260929 03:15:23.004166 16348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:37509 every 8 connection(s)
I20260929 03:15:23.005036 16218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:15:23.007113 16055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:22.994255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:56181 (local address 127.13.238.190:45251)
0929 03:15:22.994370 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:22.994374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:22.995518 (+ 1144us) server_negotiation.cc:407] Connection header received
0929 03:15:22.996553 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:22.996557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:22.996618 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:22.996674 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:22.998407 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:22.999043 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.000263 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.000629 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.000762 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:23.003649 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:23.003653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:23.003656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:23.003686 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:23.005075 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:23.005652 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:23.005656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:23.005657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:23.005749 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:23.006142 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:23.006145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:23.006147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:23.006377 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:23.006471 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:23.006892 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 03:15:23.006993 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:15:23.007953 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:22.994860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:45251 (local address 127.13.238.130:56181)
0929 03:15:22.995367 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:22.995421 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:22.996298 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:22.997088 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:22.997097 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:22.997519 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:22.998249 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:22.998270 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:22.999163 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:22.999166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:23.000142 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:23.000152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.000814 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.000819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:23.001027 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:23.001673 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:23.001688 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:23.003445 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:23.005195 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:23.005201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:23.005216 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:23.005539 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:23.005848 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:23.005851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:23.005853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:23.006033 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:23.006471 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:23.006476 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:23.006693 (+ 217us) client_negotiation.cc:769] Sending connection context
0929 03:15:23.006836 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 03:15:23.007088 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":124,"threads_started":1}
I20260929 03:15:23.009171 16349 heartbeater.cc:347] Connected to a master server at 127.13.238.190:45251
I20260929 03:15:23.009303 16349 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:23.009760 16349 heartbeater.cc:511] Master 127.13.238.190:45251 requested a full tablet report, sending...
I20260929 03:15:23.010792 16005 ts_manager.cc:194] Registered new tserver with Master: b57d5c8da10b4c4ca970f4342273a6df (127.13.238.130:37509)
I20260929 03:15:23.011507 16005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.130@KRBTEST.COM'} at 127.13.238.130:56181
I20260929 03:15:23.011611 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 16218
I20260929 03:15:23.011724 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance
I20260929 03:15:23.012043 14266 external_mini_cluster.cc:1428] Setting key de2b010d194b9f35992872566fc26f01
I20260929 03:15:23.018353 16055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:23.011254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:59653 (local address 127.13.238.190:45251)
0929 03:15:23.011351 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:23.011355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:23.011976 (+ 621us) server_negotiation.cc:407] Connection header received
0929 03:15:23.012078 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:23.012081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:23.012126 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:23.012198 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:23.012648 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.013144 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.014148 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.014390 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.014474 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:23.015494 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:23.015497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:23.015500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:23.015523 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:23.016788 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:23.017222 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:23.017224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:23.017225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:23.017288 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:23.017554 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:23.017556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:23.017557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:23.017763 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:23.017835 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:23.018119 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 03:15:23.018240 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:15:23.018424 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:23.011206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:45251 (local address 127.13.238.130:59653)
0929 03:15:23.011885 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:23.011899 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:23.012005 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:23.012229 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:23.012232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:23.012440 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:23.012555 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:23.012563 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.013262 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.013265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:23.014028 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:23.014039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.014487 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.014491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:23.014578 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:23.015018 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:23.015032 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:23.015353 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:23.016890 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:23.016893 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:23.016895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:23.017135 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:23.017373 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:23.017376 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:23.017378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:23.017470 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:23.017855 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:23.017859 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:23.017950 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 03:15:23.018073 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 03:15:23.018230 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for kudu/127.13.238.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
I20260929 03:15:23.037124 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:0
--local_ip_for_outbound_sockets=127.13.238.131
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:45251
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:23.149896 16358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:23.150236 16358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:23.150309 16358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:23.154479 16358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:23.154547 16358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:23.154603 16358 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 03:15:23.154630 16358 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:15:23.154678 16358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:15:23.160364 16358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.238.190:45251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:23.161513 16358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:23.162622 16358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:23.172286 16366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:23.172290 16363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:23.172291 16364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:23.172487 16358 server_base.cc:1034] running on GCE node
I20260929 03:15:23.172992 16358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:23.173636 16358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:23.174789 16358 hybrid_clock.cc:648] HybridClock initialized: now 1790651723174769 us; error 39 us; skew 500 ppm
Sep 29 03:15:23 dist-test-slave-nbjg krb5kdc[15961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651723, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:23.177419 16358 init.cc:377] Logged in from keytab as kudu/127.13.238.131@KRBTEST.COM (short username kudu)
I20260929 03:15:23.178305 16358 webserver.cc:460] Webserver started at http://127.13.238.131:45461/ using document root <none> and password file <none>
I20260929 03:15:23.178824 16358 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:23.178879 16358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:23.179047 16358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:23.181059 16358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-2/data/instance:
uuid: "8877d9fe57d64228a82a9fbc4f37a4cb"
format_stamp: "Formatted at 2026-09-29 03:15:23 on dist-test-slave-nbjg"
server_key: "43ed92fef361cf83ad867b3db596dba3"
server_key_iv: "2170edb037cd355ab83d984766c06c92"
server_key_version: "encryptionkey@0"
I20260929 03:15:23.181581 16358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance:
uuid: "8877d9fe57d64228a82a9fbc4f37a4cb"
format_stamp: "Formatted at 2026-09-29 03:15:23 on dist-test-slave-nbjg"
server_key: "43ed92fef361cf83ad867b3db596dba3"
server_key_iv: "2170edb037cd355ab83d984766c06c92"
server_key_version: "encryptionkey@0"
I20260929 03:15:23.185164 16358 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 03:15:23.187548 16373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:23.188755 16358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 03:15:23.188854 16358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "8877d9fe57d64228a82a9fbc4f37a4cb"
format_stamp: "Formatted at 2026-09-29 03:15:23 on dist-test-slave-nbjg"
server_key: "43ed92fef361cf83ad867b3db596dba3"
server_key_iv: "2170edb037cd355ab83d984766c06c92"
server_key_version: "encryptionkey@0"
I20260929 03:15:23.188963 16358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:23.214308 16358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:23.217113 16358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:23.217284 16358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:23.218909 16358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:23.218958 16358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:23.219002 16358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:23.219029 16358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:15:23 dist-test-slave-nbjg krb5kdc[15961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651723, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:23.228843 16358 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:38703
I20260929 03:15:23.228943 16488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:38703 every 8 connection(s)
I20260929 03:15:23.229740 16358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:15:23.232616 16055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:23.219588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:47683 (local address 127.13.238.190:45251)
0929 03:15:23.219735 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:23.219740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:23.221040 (+ 1300us) server_negotiation.cc:407] Connection header received
0929 03:15:23.222322 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:23.222325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:23.222371 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:23.222441 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:23.224004 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.224614 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.226274 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.226499 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.226580 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:23.229412 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:23.229417 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:23.229420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:23.229457 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:23.230979 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:23.231395 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:23.231397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:23.231398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:23.231472 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:23.231723 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:23.231725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:23.231726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:23.231937 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:23.232011 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:23.232402 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 03:15:23.232498 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 03:15:23.232863 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 16358
I20260929 03:15:23.232964 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance
I20260929 03:15:23.233218 14266 external_mini_cluster.cc:1428] Setting key 69c7b8d4d94be5a987ac51179fbcf189
I20260929 03:15:23.233500 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:23.220394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:45251 (local address 127.13.238.131:47683)
0929 03:15:23.220871 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:23.220943 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:23.222075 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:23.222632 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:23.222642 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:23.223133 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:23.223822 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:23.223843 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.224716 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.224720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:23.226114 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:23.226124 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.226650 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.226655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:23.226856 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:23.227493 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:23.227516 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:23.229238 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:23.231071 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:23.231075 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:23.231087 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:23.231317 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:23.231538 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:23.231540 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:23.231542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:23.231658 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:23.232013 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:23.232017 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:23.232225 (+ 208us) client_negotiation.cc:769] Sending connection context
0929 03:15:23.232366 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 03:15:23.232599 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":105,"threads_started":1}
I20260929 03:15:23.234689 16489 heartbeater.cc:347] Connected to a master server at 127.13.238.190:45251
I20260929 03:15:23.234822 16489 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:23.235343 16489 heartbeater.cc:511] Master 127.13.238.190:45251 requested a full tablet report, sending...
I20260929 03:15:23.236397 16005 ts_manager.cc:194] Registered new tserver with Master: 8877d9fe57d64228a82a9fbc4f37a4cb (127.13.238.131:38703)
I20260929 03:15:23.237097 16005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.131@KRBTEST.COM'} at 127.13.238.131:47683
I20260929 03:15:23.237425 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:15:23.243758 16055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:23.236840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:45165 (local address 127.13.238.190:45251)
0929 03:15:23.236943 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:23.236947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:23.237164 (+ 217us) server_negotiation.cc:407] Connection header received
0929 03:15:23.237315 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:23.237318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:23.237363 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:23.237430 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:23.238123 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.238679 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.239470 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.239701 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.239784 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:23.241005 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:23.241010 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:23.241012 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:23.241039 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:23.242317 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:23.242702 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:23.242705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:23.242707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:23.242781 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:23.243029 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:23.243032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:23.243034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:23.243198 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:23.243280 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:23.243522 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 03:15:23.243622 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:15:23.243770 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:23.236768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:45251 (local address 127.13.238.131:45165)
0929 03:15:23.237090 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:23.237105 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:23.237226 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:23.237689 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:23.237691 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:23.237870 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:23.238010 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:23.238016 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.238795 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.238798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:23.239361 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:23.239368 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.239801 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.239805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:23.239903 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:23.240405 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:23.240420 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:23.240853 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:23.242415 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:23.242419 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:23.242421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:23.242615 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:23.242862 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:23.242865 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:23.242867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:23.242950 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:23.243286 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:23.243289 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:23.243373 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 03:15:23.243457 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 03:15:23.243587 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":4,"thread_start_us":84,"threads_started":1}
Sep 29 03:15:23 dist-test-slave-nbjg krb5kdc[15961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:15:23 dist-test-slave-nbjg krb5kdc[15961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:23.262425 16055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:23.254815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52978 (local address 127.13.238.190:45251)
0929 03:15:23.254965 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:23.254969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:23.254981 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:15:23.255019 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:23.255021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:23.255058 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:23.255126 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:23.255814 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.256341 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.257163 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.257379 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.257466 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:23.259686 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:23.259689 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:23.259691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:23.259716 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:23.260793 (+ 1077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:23.261219 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:23.261222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:23.261223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:23.261297 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:23.261555 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:23.261559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:23.261560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:23.261733 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:23.261855 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:23.262110 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 03:15:23.262261 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 03:15:23.265173 16005 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:15:23.268142 16005 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:15:23.282426 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:23.276632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:37509 (local address 127.0.0.1:42486)
0929 03:15:23.277822 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:23.277833 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:23.277917 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:23.278372 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:23.278375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:23.278390 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:23.278464 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:23.278469 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.279942 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.279946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:23.281048 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:23.281056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.281860 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.281863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:23.281967 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:23.282003 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 03:15:23.282119 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 03:15:23.282290 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1138,"mutex_wait_us":1,"thread_start_us":105,"threads_started":1}
I20260929 03:15:23.282439 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:23.277129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42486 (local address 127.13.238.130:37509)
0929 03:15:23.277481 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:23.277493 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:23.277898 (+ 405us) server_negotiation.cc:407] Connection header received
0929 03:15:23.278022 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:23.278027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:23.278259 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:23.278397 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:23.278566 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.279727 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.281156 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.281745 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.281869 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:23.282006 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:23.282155 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 03:15:23.282273 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":96,"threads_started":1}
I20260929 03:15:23.282792 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:23.276470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37008 (local address 127.13.238.129:34855)
0929 03:15:23.278145 (+ 1675us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:23.278159 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:23.278178 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:15:23.278269 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:23.278274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:23.278444 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:23.278628 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:23.278852 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.279898 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.281089 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.281793 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.281904 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:23.282012 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:23.282576 (+ 564us) server_negotiation.cc:299] Negotiation successful
0929 03:15:23.282663 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1576,"thread_start_us":81,"threads_started":1}
I20260929 03:15:23.282644 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:23.276379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:38703 (local address 127.0.0.1:33906)
0929 03:15:23.277555 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:23.277567 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:23.277644 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:23.278179 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:23.278185 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:23.278232 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:23.278357 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:23.278364 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.279486 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.279489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:23.280809 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:23.280820 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.282075 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.282079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:23.282316 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:23.282351 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 03:15:23.282419 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 03:15:23.282503 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1119,"mutex_wait_us":105,"thread_start_us":85,"threads_started":1}
I20260929 03:15:23.282770 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:23.276335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:34855 (local address 127.0.0.1:37008)
0929 03:15:23.276965 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:23.277014 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:23.277135 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:23.278616 (+ 1481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:23.278619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:23.278637 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:23.278735 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:23.278741 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.280029 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.280033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:23.280935 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:23.280945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.282075 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.282079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:23.282311 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:23.282352 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 03:15:23.282419 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 03:15:23.282503 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":54,"thread_start_us":71,"threads_started":1}
I20260929 03:15:23.283087 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:23.277161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33906 (local address 127.13.238.131:38703)
0929 03:15:23.277654 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:23.277664 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:23.277731 (+ 67us) server_negotiation.cc:407] Connection header received
0929 03:15:23.277856 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:23.277860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:23.278031 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:23.278173 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:23.278497 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.279376 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.280942 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.281751 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.281888 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:23.282013 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:23.282847 (+ 834us) server_negotiation.cc:299] Negotiation successful
0929 03:15:23.282937 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":116,"threads_started":1}
I20260929 03:15:23.285626 16283 tablet_service.cc:1467] Processing CreateTablet for tablet 28993c0cc3c34c36b1679dcc59e299fa (DEFAULT_TABLE table=test-table [id=a56923caf1124eb9ac99f48b8a2d9eb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:23.285727 16143 tablet_service.cc:1467] Processing CreateTablet for tablet 28993c0cc3c34c36b1679dcc59e299fa (DEFAULT_TABLE table=test-table [id=a56923caf1124eb9ac99f48b8a2d9eb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:23.285727 16423 tablet_service.cc:1467] Processing CreateTablet for tablet 28993c0cc3c34c36b1679dcc59e299fa (DEFAULT_TABLE table=test-table [id=a56923caf1124eb9ac99f48b8a2d9eb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:23.286620 16283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28993c0cc3c34c36b1679dcc59e299fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:23.286726 16423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28993c0cc3c34c36b1679dcc59e299fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:23.286720 16143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28993c0cc3c34c36b1679dcc59e299fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:23.293457 16512 tablet_bootstrap.cc:492] T 28993c0cc3c34c36b1679dcc59e299fa P b57d5c8da10b4c4ca970f4342273a6df: Bootstrap starting.
I20260929 03:15:23.295001 16513 tablet_bootstrap.cc:492] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522: Bootstrap starting.
I20260929 03:15:23.296061 16512 tablet_bootstrap.cc:654] T 28993c0cc3c34c36b1679dcc59e299fa P b57d5c8da10b4c4ca970f4342273a6df: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:23.296774 16513 tablet_bootstrap.cc:654] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:23.296793 16512 log.cc:824] T 28993c0cc3c34c36b1679dcc59e299fa P b57d5c8da10b4c4ca970f4342273a6df: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:23.297403 16514 tablet_bootstrap.cc:492] T 28993c0cc3c34c36b1679dcc59e299fa P 8877d9fe57d64228a82a9fbc4f37a4cb: Bootstrap starting.
I20260929 03:15:23.297519 16513 log.cc:824] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:23.298888 16512 tablet_bootstrap.cc:492] T 28993c0cc3c34c36b1679dcc59e299fa P b57d5c8da10b4c4ca970f4342273a6df: No bootstrap required, opened a new log
I20260929 03:15:23.299080 16512 ts_tablet_manager.cc:1397] T 28993c0cc3c34c36b1679dcc59e299fa P b57d5c8da10b4c4ca970f4342273a6df: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 03:15:23.299460 16513 tablet_bootstrap.cc:492] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522: No bootstrap required, opened a new log
I20260929 03:15:23.299631 16513 ts_tablet_manager.cc:1397] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 03:15:23.299685 16514 tablet_bootstrap.cc:654] T 28993c0cc3c34c36b1679dcc59e299fa P 8877d9fe57d64228a82a9fbc4f37a4cb: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:23.300324 16514 log.cc:824] T 28993c0cc3c34c36b1679dcc59e299fa P 8877d9fe57d64228a82a9fbc4f37a4cb: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:23.301893 16514 tablet_bootstrap.cc:492] T 28993c0cc3c34c36b1679dcc59e299fa P 8877d9fe57d64228a82a9fbc4f37a4cb: No bootstrap required, opened a new log
I20260929 03:15:23.302057 16514 ts_tablet_manager.cc:1397] T 28993c0cc3c34c36b1679dcc59e299fa P 8877d9fe57d64228a82a9fbc4f37a4cb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 03:15:23.303005 16512 raft_consensus.cc:348] T 28993c0cc3c34c36b1679dcc59e299fa P b57d5c8da10b4c4ca970f4342273a6df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } }
I20260929 03:15:23.303282 16512 raft_consensus.cc:374] T 28993c0cc3c34c36b1679dcc59e299fa P b57d5c8da10b4c4ca970f4342273a6df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:23.303336 16512 raft_consensus.cc:729] T 28993c0cc3c34c36b1679dcc59e299fa P b57d5c8da10b4c4ca970f4342273a6df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57d5c8da10b4c4ca970f4342273a6df, State: Initialized, Role: FOLLOWER
I20260929 03:15:23.303207 16513 raft_consensus.cc:348] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } }
I20260929 03:15:23.303416 16513 raft_consensus.cc:374] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:23.303475 16513 raft_consensus.cc:729] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c9b446b29f4e8f966a0c34278bc522, State: Initialized, Role: FOLLOWER
I20260929 03:15:23.303895 16512 consensus_queue.cc:260] T 28993c0cc3c34c36b1679dcc59e299fa P b57d5c8da10b4c4ca970f4342273a6df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } }
I20260929 03:15:23.303895 16513 consensus_queue.cc:260] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } }
I20260929 03:15:23.304593 16209 heartbeater.cc:503] Master 127.13.238.190:45251 was elected leader, sending a full tablet report...
I20260929 03:15:23.304865 16513 ts_tablet_manager.cc:1428] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 03:15:23.304862 16512 ts_tablet_manager.cc:1428] T 28993c0cc3c34c36b1679dcc59e299fa P b57d5c8da10b4c4ca970f4342273a6df: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 03:15:23.305200 16349 heartbeater.cc:503] Master 127.13.238.190:45251 was elected leader, sending a full tablet report...
I20260929 03:15:23.307075 16514 raft_consensus.cc:348] T 28993c0cc3c34c36b1679dcc59e299fa P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } }
I20260929 03:15:23.307343 16514 raft_consensus.cc:374] T 28993c0cc3c34c36b1679dcc59e299fa P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:23.307442 16514 raft_consensus.cc:729] T 28993c0cc3c34c36b1679dcc59e299fa P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8877d9fe57d64228a82a9fbc4f37a4cb, State: Initialized, Role: FOLLOWER
I20260929 03:15:23.308372 16514 consensus_queue.cc:260] T 28993c0cc3c34c36b1679dcc59e299fa P 8877d9fe57d64228a82a9fbc4f37a4cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } }
I20260929 03:15:23.309274 16489 heartbeater.cc:503] Master 127.13.238.190:45251 was elected leader, sending a full tablet report...
I20260929 03:15:23.309370 16514 ts_tablet_manager.cc:1428] T 28993c0cc3c34c36b1679dcc59e299fa P 8877d9fe57d64228a82a9fbc4f37a4cb: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260929 03:15:23.480942 16490 tablet.cc:2369] T 28993c0cc3c34c36b1679dcc59e299fa P 8877d9fe57d64228a82a9fbc4f37a4cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:15:23.506170 16350 tablet.cc:2369] T 28993c0cc3c34c36b1679dcc59e299fa P b57d5c8da10b4c4ca970f4342273a6df: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:15:23.540337 16210 tablet.cc:2369] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:15:23.588809 16518 raft_consensus.cc:482] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:15:23.589008 16518 raft_consensus.cc:504] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } }
I20260929 03:15:23.590240 16518 leader_election.cc:288] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8877d9fe57d64228a82a9fbc4f37a4cb (127.13.238.131:38703), b57d5c8da10b4c4ca970f4342273a6df (127.13.238.130:37509)
I20260929 03:15:23.594314 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:23.590503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:38703 (local address 127.13.238.129:40785)
0929 03:15:23.590979 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:23.590993 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:23.591107 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:23.591359 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:23.591361 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:23.591375 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:23.591467 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:23.591472 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.592232 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.592235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:23.593208 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:23.593218 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.593884 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.593888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:23.593980 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:23.594000 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 03:15:23.594070 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 03:15:23.594139 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":120,"thread_start_us":71,"threads_started":1}
I20260929 03:15:23.594301 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:23.590618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:40785 (local address 127.13.238.131:38703)
0929 03:15:23.590752 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:23.590758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:23.591098 (+ 340us) server_negotiation.cc:407] Connection header received
0929 03:15:23.591214 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:23.591218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:23.591268 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:23.591338 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:23.591582 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.592123 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.593323 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.593771 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.593891 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:23.593939 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:23.594093 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 03:15:23.594158 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 03:15:23.594379 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:23.590691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:57535 (local address 127.13.238.130:37509)
0929 03:15:23.590800 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:23.590804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:23.591000 (+ 196us) server_negotiation.cc:407] Connection header received
0929 03:15:23.591095 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:23.591098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:23.591144 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:23.591212 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:23.591514 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.592104 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.593318 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.593740 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.593849 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:23.593897 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:23.594176 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 03:15:23.594278 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:15:23.594383 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:23.590503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:37509 (local address 127.13.238.129:57535)
0929 03:15:23.590895 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:23.590908 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:23.591011 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:23.591237 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:23.591240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:23.591254 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:23.591409 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:23.591417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.592208 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.592211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:23.593209 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:23.593218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.593946 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.593949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:23.594024 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:23.594040 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 03:15:23.594137 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 03:15:23.594229 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":95,"thread_start_us":102,"threads_started":1}
I20260929 03:15:23.595160 16443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28993c0cc3c34c36b1679dcc59e299fa" candidate_uuid: "10c9b446b29f4e8f966a0c34278bc522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" is_pre_election: true
I20260929 03:15:23.595155 16303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28993c0cc3c34c36b1679dcc59e299fa" candidate_uuid: "10c9b446b29f4e8f966a0c34278bc522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b57d5c8da10b4c4ca970f4342273a6df" is_pre_election: true
I20260929 03:15:23.595476 16303 raft_consensus.cc:2454] T 28993c0cc3c34c36b1679dcc59e299fa P b57d5c8da10b4c4ca970f4342273a6df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c9b446b29f4e8f966a0c34278bc522 in term 0.
I20260929 03:15:23.595476 16443 raft_consensus.cc:2454] T 28993c0cc3c34c36b1679dcc59e299fa P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c9b446b29f4e8f966a0c34278bc522 in term 0.
I20260929 03:15:23.595956 16095 leader_election.cc:302] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10c9b446b29f4e8f966a0c34278bc522, 8877d9fe57d64228a82a9fbc4f37a4cb; no voters:
I20260929 03:15:23.596228 16518 raft_consensus.cc:2781] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:15:23.596306 16518 raft_consensus.cc:482] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:15:23.596371 16518 raft_consensus.cc:3037] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:23.597435 16518 raft_consensus.cc:504] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } }
I20260929 03:15:23.597779 16518 leader_election.cc:288] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522 [CANDIDATE]: Term 1 election: Requested vote from peers 8877d9fe57d64228a82a9fbc4f37a4cb (127.13.238.131:38703), b57d5c8da10b4c4ca970f4342273a6df (127.13.238.130:37509)
I20260929 03:15:23.598109 16303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28993c0cc3c34c36b1679dcc59e299fa" candidate_uuid: "10c9b446b29f4e8f966a0c34278bc522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b57d5c8da10b4c4ca970f4342273a6df"
I20260929 03:15:23.598266 16443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28993c0cc3c34c36b1679dcc59e299fa" candidate_uuid: "10c9b446b29f4e8f966a0c34278bc522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb"
I20260929 03:15:23.598266 16303 raft_consensus.cc:3037] T 28993c0cc3c34c36b1679dcc59e299fa P b57d5c8da10b4c4ca970f4342273a6df [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:23.598383 16443 raft_consensus.cc:3037] T 28993c0cc3c34c36b1679dcc59e299fa P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:23.599537 16303 raft_consensus.cc:2454] T 28993c0cc3c34c36b1679dcc59e299fa P b57d5c8da10b4c4ca970f4342273a6df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c9b446b29f4e8f966a0c34278bc522 in term 1.
I20260929 03:15:23.599871 16443 raft_consensus.cc:2454] T 28993c0cc3c34c36b1679dcc59e299fa P 8877d9fe57d64228a82a9fbc4f37a4cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c9b446b29f4e8f966a0c34278bc522 in term 1.
I20260929 03:15:23.599891 16097 leader_election.cc:302] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10c9b446b29f4e8f966a0c34278bc522, b57d5c8da10b4c4ca970f4342273a6df; no voters:
I20260929 03:15:23.600044 16518 raft_consensus.cc:2781] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:23.600287 16518 raft_consensus.cc:686] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522 [term 1 LEADER]: Becoming Leader. State: Replica: 10c9b446b29f4e8f966a0c34278bc522, State: Running, Role: LEADER
I20260929 03:15:23.600623 16518 consensus_queue.cc:237] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } }
I20260929 03:15:23.603688 16006 catalog_manager.cc:5495] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10c9b446b29f4e8f966a0c34278bc522 (127.13.238.129). New cstate: current_term: 1 leader_uuid: "10c9b446b29f4e8f966a0c34278bc522" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } health_report { overall_health: UNKNOWN } } }
Sep 29 03:15:23 dist-test-slave-nbjg krb5kdc[15961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651722, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:23.687602 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:23.679960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:45251 (local address 127.0.0.1:52986)
0929 03:15:23.680258 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:23.680273 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:23.680379 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:23.680604 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:23.680606 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:23.681012 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:23.681106 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:23.681112 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.681828 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.681830 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:23.682453 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:23.682462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.682836 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.682838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:23.682894 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:23.683468 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:23.683480 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:23.684830 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:23.686269 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:23.686277 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:23.686283 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:23.686500 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:23.686714 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:23.686716 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:23.686717 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:23.686784 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:23.687105 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:23.687112 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:23.687232 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 03:15:23.687324 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 03:15:23.687442 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":90,"threads_started":1}
I20260929 03:15:23.687629 16055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:23.680022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52986 (local address 127.13.238.190:45251)
0929 03:15:23.680138 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:23.680141 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:23.680339 (+ 198us) server_negotiation.cc:407] Connection header received
0929 03:15:23.680450 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:23.680452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:23.680502 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:23.680576 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:23.681221 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.681746 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.682539 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:23.682748 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:23.682820 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:23.684954 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:23.684958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:23.684960 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:23.684982 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:23.686142 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:23.686574 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:23.686577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:23.686578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:23.686648 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:23.686847 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:23.686849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:23.686850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:23.687014 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:23.687084 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:23.687367 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 03:15:23.687506 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":41}
I20260929 03:15:23.691428 16006 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:52986:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 03:15:23.692152 16006 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:15:23.697084 16143 tablet_service.cc:1467] Processing CreateTablet for tablet ab4f3be62e134910845a66388a3bc3b1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c7370466f7046a390d37d1d02f56dc9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:15:23.697187 16423 tablet_service.cc:1467] Processing CreateTablet for tablet ab4f3be62e134910845a66388a3bc3b1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c7370466f7046a390d37d1d02f56dc9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:15:23.697199 16283 tablet_service.cc:1467] Processing CreateTablet for tablet ab4f3be62e134910845a66388a3bc3b1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c7370466f7046a390d37d1d02f56dc9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:15:23.697759 16143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab4f3be62e134910845a66388a3bc3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:23.697880 16423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab4f3be62e134910845a66388a3bc3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:23.697880 16283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab4f3be62e134910845a66388a3bc3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:23.700313 16513 tablet_bootstrap.cc:492] T ab4f3be62e134910845a66388a3bc3b1 P 10c9b446b29f4e8f966a0c34278bc522: Bootstrap starting.
I20260929 03:15:23.700670 16512 tablet_bootstrap.cc:492] T ab4f3be62e134910845a66388a3bc3b1 P b57d5c8da10b4c4ca970f4342273a6df: Bootstrap starting.
I20260929 03:15:23.700891 16514 tablet_bootstrap.cc:492] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb: Bootstrap starting.
I20260929 03:15:23.701565 16513 tablet_bootstrap.cc:654] T ab4f3be62e134910845a66388a3bc3b1 P 10c9b446b29f4e8f966a0c34278bc522: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:23.701565 16512 tablet_bootstrap.cc:654] T ab4f3be62e134910845a66388a3bc3b1 P b57d5c8da10b4c4ca970f4342273a6df: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:23.701695 16514 tablet_bootstrap.cc:654] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:23.702816 16513 tablet_bootstrap.cc:492] T ab4f3be62e134910845a66388a3bc3b1 P 10c9b446b29f4e8f966a0c34278bc522: No bootstrap required, opened a new log
I20260929 03:15:23.702832 16514 tablet_bootstrap.cc:492] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb: No bootstrap required, opened a new log
I20260929 03:15:23.702852 16512 tablet_bootstrap.cc:492] T ab4f3be62e134910845a66388a3bc3b1 P b57d5c8da10b4c4ca970f4342273a6df: No bootstrap required, opened a new log
I20260929 03:15:23.702894 16513 ts_tablet_manager.cc:1397] T ab4f3be62e134910845a66388a3bc3b1 P 10c9b446b29f4e8f966a0c34278bc522: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:15:23.702905 16514 ts_tablet_manager.cc:1397] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:15:23.702913 16512 ts_tablet_manager.cc:1397] T ab4f3be62e134910845a66388a3bc3b1 P b57d5c8da10b4c4ca970f4342273a6df: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:15:23.703265 16513 raft_consensus.cc:348] T ab4f3be62e134910845a66388a3bc3b1 P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:23.703344 16513 raft_consensus.cc:374] T ab4f3be62e134910845a66388a3bc3b1 P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:23.703364 16513 raft_consensus.cc:729] T ab4f3be62e134910845a66388a3bc3b1 P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c9b446b29f4e8f966a0c34278bc522, State: Initialized, Role: FOLLOWER
I20260929 03:15:23.703327 16512 raft_consensus.cc:348] T ab4f3be62e134910845a66388a3bc3b1 P b57d5c8da10b4c4ca970f4342273a6df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:23.703327 16514 raft_consensus.cc:348] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:23.703425 16512 raft_consensus.cc:374] T ab4f3be62e134910845a66388a3bc3b1 P b57d5c8da10b4c4ca970f4342273a6df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:23.703425 16514 raft_consensus.cc:374] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:23.703456 16514 raft_consensus.cc:729] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8877d9fe57d64228a82a9fbc4f37a4cb, State: Initialized, Role: FOLLOWER
I20260929 03:15:23.703456 16512 raft_consensus.cc:729] T ab4f3be62e134910845a66388a3bc3b1 P b57d5c8da10b4c4ca970f4342273a6df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57d5c8da10b4c4ca970f4342273a6df, State: Initialized, Role: FOLLOWER
I20260929 03:15:23.703460 16513 consensus_queue.cc:260] T ab4f3be62e134910845a66388a3bc3b1 P 10c9b446b29f4e8f966a0c34278bc522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:23.703572 16514 consensus_queue.cc:260] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:23.703572 16512 consensus_queue.cc:260] T ab4f3be62e134910845a66388a3bc3b1 P b57d5c8da10b4c4ca970f4342273a6df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:23.703691 16513 ts_tablet_manager.cc:1428] T ab4f3be62e134910845a66388a3bc3b1 P 10c9b446b29f4e8f966a0c34278bc522: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:23.703810 16514 ts_tablet_manager.cc:1428] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 03:15:23.703817 16512 ts_tablet_manager.cc:1428] T ab4f3be62e134910845a66388a3bc3b1 P b57d5c8da10b4c4ca970f4342273a6df: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 03:15:23.703835 16519 tablet_replica.cc:439] T ab4f3be62e134910845a66388a3bc3b1 P b57d5c8da10b4c4ca970f4342273a6df: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } } }
I20260929 03:15:23.703986 16519 tablet_replica.cc:442] T ab4f3be62e134910845a66388a3bc3b1 P b57d5c8da10b4c4ca970f4342273a6df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:15:23.704188 16520 tablet_replica.cc:439] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } } }
I20260929 03:15:23.704316 16520 tablet_replica.cc:442] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:15:23.704604 16518 tablet_replica.cc:439] T ab4f3be62e134910845a66388a3bc3b1 P 10c9b446b29f4e8f966a0c34278bc522: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } } }
I20260929 03:15:23.704734 16518 tablet_replica.cc:442] T ab4f3be62e134910845a66388a3bc3b1 P 10c9b446b29f4e8f966a0c34278bc522: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:15:24.011814 16520 raft_consensus.cc:482] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:15:24.011973 16520 raft_consensus.cc:504] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:24.013125 16520 leader_election.cc:288] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10c9b446b29f4e8f966a0c34278bc522 (127.13.238.129:34855), b57d5c8da10b4c4ca970f4342273a6df (127.13.238.130:37509)
I20260929 03:15:24.017045 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:24.013580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:46501 (local address 127.13.238.129:34855)
0929 03:15:24.013838 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:24.013855 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:24.013872 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:15:24.014022 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:24.014024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:24.014073 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:24.014167 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:24.014472 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.015006 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.016080 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.016483 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.016595 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:24.016650 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:24.016841 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 03:15:24.016918 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":87,"threads_started":1}
I20260929 03:15:24.017048 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:24.013472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:34487 (local address 127.13.238.130:37509)
0929 03:15:24.013639 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:24.013643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:24.014104 (+ 461us) server_negotiation.cc:407] Connection header received
0929 03:15:24.014158 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:24.014160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:24.014223 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:24.014288 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:24.014583 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.015067 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.016086 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.016478 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.016594 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:24.016650 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:24.016863 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 03:15:24.016937 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 03:15:24.017056 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:24.013390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:34855 (local address 127.13.238.131:46501)
0929 03:15:24.013758 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:24.013774 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:24.013887 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:24.014213 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:24.014216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:24.014236 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:24.014362 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:24.014369 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.015122 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.015125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:24.015937 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:24.015946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.016595 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.016599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:24.016717 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:24.016736 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 03:15:24.016799 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 03:15:24.016880 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":2,"thread_start_us":97,"threads_started":1}
I20260929 03:15:24.017063 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:24.013393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:37509 (local address 127.13.238.131:34487)
0929 03:15:24.014011 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:24.014022 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:24.014095 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:24.014339 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:24.014342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:24.014362 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:24.014464 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:24.014470 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.015175 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.015178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:24.015947 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:24.015956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.016595 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.016599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:24.016740 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:24.016759 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 03:15:24.016825 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 03:15:24.016898 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":365,"thread_start_us":51,"threads_started":1}
I20260929 03:15:24.017644 16303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab4f3be62e134910845a66388a3bc3b1" candidate_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b57d5c8da10b4c4ca970f4342273a6df" is_pre_election: true
I20260929 03:15:24.017819 16303 raft_consensus.cc:2454] T ab4f3be62e134910845a66388a3bc3b1 P b57d5c8da10b4c4ca970f4342273a6df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8877d9fe57d64228a82a9fbc4f37a4cb in term 0.
I20260929 03:15:24.017869 16163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab4f3be62e134910845a66388a3bc3b1" candidate_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c9b446b29f4e8f966a0c34278bc522" is_pre_election: true
I20260929 03:15:24.018126 16163 raft_consensus.cc:2454] T ab4f3be62e134910845a66388a3bc3b1 P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8877d9fe57d64228a82a9fbc4f37a4cb in term 0.
I20260929 03:15:24.018296 16377 leader_election.cc:302] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8877d9fe57d64228a82a9fbc4f37a4cb, b57d5c8da10b4c4ca970f4342273a6df; no voters:
I20260929 03:15:24.018610 16520 raft_consensus.cc:2781] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:15:24.018682 16520 raft_consensus.cc:482] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:15:24.018724 16520 raft_consensus.cc:3037] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:24.019727 16520 raft_consensus.cc:504] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:24.020117 16520 leader_election.cc:288] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb [CANDIDATE]: Term 1 election: Requested vote from peers 10c9b446b29f4e8f966a0c34278bc522 (127.13.238.129:34855), b57d5c8da10b4c4ca970f4342273a6df (127.13.238.130:37509)
I20260929 03:15:24.020519 16163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab4f3be62e134910845a66388a3bc3b1" candidate_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c9b446b29f4e8f966a0c34278bc522"
I20260929 03:15:24.020545 16303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab4f3be62e134910845a66388a3bc3b1" candidate_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b57d5c8da10b4c4ca970f4342273a6df"
I20260929 03:15:24.020699 16163 raft_consensus.cc:3037] T ab4f3be62e134910845a66388a3bc3b1 P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:24.020717 16303 raft_consensus.cc:3037] T ab4f3be62e134910845a66388a3bc3b1 P b57d5c8da10b4c4ca970f4342273a6df [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:24.021699 16163 raft_consensus.cc:2454] T ab4f3be62e134910845a66388a3bc3b1 P 10c9b446b29f4e8f966a0c34278bc522 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8877d9fe57d64228a82a9fbc4f37a4cb in term 1.
I20260929 03:15:24.021742 16303 raft_consensus.cc:2454] T ab4f3be62e134910845a66388a3bc3b1 P b57d5c8da10b4c4ca970f4342273a6df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8877d9fe57d64228a82a9fbc4f37a4cb in term 1.
I20260929 03:15:24.022029 16374 leader_election.cc:302] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10c9b446b29f4e8f966a0c34278bc522, 8877d9fe57d64228a82a9fbc4f37a4cb; no voters:
I20260929 03:15:24.022240 16520 raft_consensus.cc:2781] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:24.022440 16520 raft_consensus.cc:686] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb [term 1 LEADER]: Becoming Leader. State: Replica: 8877d9fe57d64228a82a9fbc4f37a4cb, State: Running, Role: LEADER
I20260929 03:15:24.022804 16520 consensus_queue.cc:237] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:24.025202 16537 tablet_replica.cc:439] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb: TxnStatusTablet state changed. Reason: New leader 8877d9fe57d64228a82a9fbc4f37a4cb. Latest consensus state: current_term: 1 leader_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } } }
I20260929 03:15:24.025321 16537 tablet_replica.cc:442] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:15:24.025753 16539 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 03:15:24.025954 16539 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 03:15:24.026345 16006 catalog_manager.cc:5495] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8877d9fe57d64228a82a9fbc4f37a4cb (127.13.238.131). New cstate: current_term: 1 leader_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } health_report { overall_health: HEALTHY } } }
I20260929 03:15:24.071969 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:24.068301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33912 (local address 127.13.238.131:38703)
0929 03:15:24.068439 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:24.068444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:24.068457 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:15:24.068678 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:24.068681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:24.068737 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:24.068822 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:24.069080 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.069584 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.070487 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.070709 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.070797 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:24.071105 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:24.071570 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:24.071677 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:24.071767 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 03:15:24.071818 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 03:15:24.071969 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:24.068144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:38703 (local address 127.0.0.1:33912)
0929 03:15:24.068390 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:24.068405 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:24.068524 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:24.068855 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:24.068858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:24.068865 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:15:24.068972 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:24.068979 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.069695 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.069699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:24.070374 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:24.070384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.070815 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.070818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:24.070908 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:24.070956 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:15:24.071675 (+ 719us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:15:24.071691 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 03:15:24.071756 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 03:15:24.071843 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260929 03:15:24.077301 16163 raft_consensus.cc:1261] T ab4f3be62e134910845a66388a3bc3b1 P 10c9b446b29f4e8f966a0c34278bc522 [term 1 FOLLOWER]: Refusing update from remote peer 8877d9fe57d64228a82a9fbc4f37a4cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:24.077301 16303 raft_consensus.cc:1261] T ab4f3be62e134910845a66388a3bc3b1 P b57d5c8da10b4c4ca970f4342273a6df [term 1 FOLLOWER]: Refusing update from remote peer 8877d9fe57d64228a82a9fbc4f37a4cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:24.078016 16520 consensus_queue.cc:1035] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:24.078222 16537 consensus_queue.cc:1035] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:24.081566 16518 tablet_replica.cc:439] T ab4f3be62e134910845a66388a3bc3b1 P 10c9b446b29f4e8f966a0c34278bc522: TxnStatusTablet state changed. Reason: New leader 8877d9fe57d64228a82a9fbc4f37a4cb. Latest consensus state: current_term: 1 leader_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } } }
I20260929 03:15:24.081689 16518 tablet_replica.cc:442] T ab4f3be62e134910845a66388a3bc3b1 P 10c9b446b29f4e8f966a0c34278bc522: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:15:24.082018 16519 tablet_replica.cc:439] T ab4f3be62e134910845a66388a3bc3b1 P b57d5c8da10b4c4ca970f4342273a6df: TxnStatusTablet state changed. Reason: New leader 8877d9fe57d64228a82a9fbc4f37a4cb. Latest consensus state: current_term: 1 leader_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } } }
I20260929 03:15:24.082125 16519 tablet_replica.cc:442] T ab4f3be62e134910845a66388a3bc3b1 P b57d5c8da10b4c4ca970f4342273a6df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:15:24.085204 16537 tablet_replica.cc:439] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } } }
I20260929 03:15:24.085332 16537 tablet_replica.cc:442] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:15:24.085456 16537 tablet_replica.cc:439] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } } }
I20260929 03:15:24.085530 16537 tablet_replica.cc:442] T ab4f3be62e134910845a66388a3bc3b1 P 8877d9fe57d64228a82a9fbc4f37a4cb: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:15:24.085605 16519 tablet_replica.cc:439] T ab4f3be62e134910845a66388a3bc3b1 P b57d5c8da10b4c4ca970f4342273a6df: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } } }
I20260929 03:15:24.085697 16519 tablet_replica.cc:442] T ab4f3be62e134910845a66388a3bc3b1 P b57d5c8da10b4c4ca970f4342273a6df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:15:24.086907 16543 mvcc.cc:205] Tried to move back new op lower bound from 7334509461813587968 to 7334509461599981568. Current Snapshot: MvccSnapshot[applied={T|T < 7334509461813587968}]
I20260929 03:15:24.086966 16518 tablet_replica.cc:439] T ab4f3be62e134910845a66388a3bc3b1 P 10c9b446b29f4e8f966a0c34278bc522: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } } }
I20260929 03:15:24.087098 16518 tablet_replica.cc:442] T ab4f3be62e134910845a66388a3bc3b1 P 10c9b446b29f4e8f966a0c34278bc522: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:15:24.089515 16518 consensus_queue.cc:1035] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:24.095306 16523 consensus_queue.cc:1035] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:24.095724 16544 mvcc.cc:205] Tried to move back new op lower bound from 7334509461813587968 to 7334509461599981568. Current Snapshot: MvccSnapshot[applied={T|T < 7334509461813587968}]
I20260929 03:15:24.100318 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:24.094804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37020 (local address 127.13.238.129:34855)
0929 03:15:24.094920 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:24.094925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:24.096197 (+ 1272us) server_negotiation.cc:407] Connection header received
0929 03:15:24.096382 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:24.096385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:24.096448 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:24.096545 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:24.096917 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.097610 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.098904 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.099213 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.099301 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:24.099578 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:24.099976 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:24.100078 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:24.100152 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 03:15:24.100201 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 03:15:24.110749 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:24.107252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:38703 (local address 127.13.238.129:59713)
0929 03:15:24.107534 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:24.107546 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:24.107631 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:24.107835 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:24.107837 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:24.107843 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:15:24.107930 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:24.107934 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.108917 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.108920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:24.109528 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:24.109536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.110062 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.110065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:24.110134 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:24.110178 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:15:24.110506 (+ 328us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:15:24.110511 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:15:24.110553 (+ 42us) client_negotiation.cc:240] Negotiation successful
0929 03:15:24.110618 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":88,"threads_started":1}
I20260929 03:15:24.110750 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:24.107361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:59713 (local address 127.13.238.131:38703)
0929 03:15:24.107479 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:24.107484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:24.107610 (+ 126us) server_negotiation.cc:407] Connection header received
0929 03:15:24.107701 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:24.107705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:24.107753 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:24.107829 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:24.108014 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.108826 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.109632 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.109964 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.110061 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:24.110353 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:24.110433 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:24.110500 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:24.110565 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 03:15:24.110618 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:15:24.137989 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:24.134027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:56413 (local address 127.13.238.129:34855)
0929 03:15:24.134144 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:24.134148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:24.134443 (+ 295us) server_negotiation.cc:407] Connection header received
0929 03:15:24.134586 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:24.134589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:24.134645 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:24.134715 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:24.135004 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.135654 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.136921 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.137151 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.137226 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:24.137560 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:24.137625 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:24.137671 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:24.137797 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 03:15:24.137840 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 03:15:24.138038 16561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:24.133915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:34855 (local address 127.13.238.131:56413)
0929 03:15:24.134309 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:24.134325 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:24.134487 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:24.134733 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:24.134736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:24.134743 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:15:24.134877 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:24.134883 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.135781 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.135786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:24.136792 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:24.136805 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.137276 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.137280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:24.137385 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:24.137441 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:15:24.137705 (+ 264us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:15:24.137711 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:15:24.137787 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 03:15:24.137871 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":147,"threads_started":1}
I20260929 03:15:24.163952 16006 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:24.164172 16006 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:24.167050 16006 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc: Sending DeleteTablet for 3 replicas of tablet 28993c0cc3c34c36b1679dcc59e299fa
I20260929 03:15:24.168030 16143 tablet_service.cc:1514] Processing DeleteTablet for tablet 28993c0cc3c34c36b1679dcc59e299fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:24 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:37008
I20260929 03:15:24.168118 16283 tablet_service.cc:1514] Processing DeleteTablet for tablet 28993c0cc3c34c36b1679dcc59e299fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:24 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:42486
I20260929 03:15:24.168710 16565 tablet_replica.cc:330] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522: stopping tablet replica
I20260929 03:15:24.169016 16565 raft_consensus.cc:2229] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:15:24.169374 16565 raft_consensus.cc:2258] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:24.169863 16006 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:15:24.170349 16006 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:15:24.170746 16565 ts_tablet_manager.cc:1905] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:24.171382 16423 tablet_service.cc:1514] Processing DeleteTablet for tablet 28993c0cc3c34c36b1679dcc59e299fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:24 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:33906
I20260929 03:15:24.172003 16566 tablet_replica.cc:330] T 28993c0cc3c34c36b1679dcc59e299fa P b57d5c8da10b4c4ca970f4342273a6df: stopping tablet replica
I20260929 03:15:24.172358 16566 raft_consensus.cc:2229] T 28993c0cc3c34c36b1679dcc59e299fa P b57d5c8da10b4c4ca970f4342273a6df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:15:24.172640 16566 raft_consensus.cc:2258] T 28993c0cc3c34c36b1679dcc59e299fa P b57d5c8da10b4c4ca970f4342273a6df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:24.174762 16566 ts_tablet_manager.cc:1905] T 28993c0cc3c34c36b1679dcc59e299fa P b57d5c8da10b4c4ca970f4342273a6df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:24.176414 16143 tablet_service.cc:1467] Processing CreateTablet for tablet c2bb0a40f1884b1ba865537d32d5b1d9 (DEFAULT_TABLE table=test-table [id=ccdba4a7748b4a1c9e6a7524d1193b45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:24.176684 16143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2bb0a40f1884b1ba865537d32d5b1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:24.176976 16567 tablet_replica.cc:330] T 28993c0cc3c34c36b1679dcc59e299fa P 8877d9fe57d64228a82a9fbc4f37a4cb: stopping tablet replica
I20260929 03:15:24.177304 16567 raft_consensus.cc:2229] T 28993c0cc3c34c36b1679dcc59e299fa P 8877d9fe57d64228a82a9fbc4f37a4cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:15:24.177491 16567 raft_consensus.cc:2258] T 28993c0cc3c34c36b1679dcc59e299fa P 8877d9fe57d64228a82a9fbc4f37a4cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:24.178397 16566 ts_tablet_manager.cc:1918] T 28993c0cc3c34c36b1679dcc59e299fa P b57d5c8da10b4c4ca970f4342273a6df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:15:24.178540 16566 log.cc:1196] T 28993c0cc3c34c36b1679dcc59e299fa P b57d5c8da10b4c4ca970f4342273a6df: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-1/wal/wals/28993c0cc3c34c36b1679dcc59e299fa
I20260929 03:15:24.178907 16566 ts_tablet_manager.cc:1939] T 28993c0cc3c34c36b1679dcc59e299fa P b57d5c8da10b4c4ca970f4342273a6df: Deleting consensus metadata
I20260929 03:15:24.179493 16283 tablet_service.cc:1467] Processing CreateTablet for tablet c2bb0a40f1884b1ba865537d32d5b1d9 (DEFAULT_TABLE table=test-table [id=ccdba4a7748b4a1c9e6a7524d1193b45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:24.179752 16283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2bb0a40f1884b1ba865537d32d5b1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:24.182025 16513 tablet_bootstrap.cc:492] T c2bb0a40f1884b1ba865537d32d5b1d9 P 10c9b446b29f4e8f966a0c34278bc522: Bootstrap starting.
I20260929 03:15:24.183074 16513 tablet_bootstrap.cc:654] T c2bb0a40f1884b1ba865537d32d5b1d9 P 10c9b446b29f4e8f966a0c34278bc522: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:24.183502 16567 ts_tablet_manager.cc:1905] T 28993c0cc3c34c36b1679dcc59e299fa P 8877d9fe57d64228a82a9fbc4f37a4cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:24.183766 16512 tablet_bootstrap.cc:492] T c2bb0a40f1884b1ba865537d32d5b1d9 P b57d5c8da10b4c4ca970f4342273a6df: Bootstrap starting.
I20260929 03:15:24.184759 16512 tablet_bootstrap.cc:654] T c2bb0a40f1884b1ba865537d32d5b1d9 P b57d5c8da10b4c4ca970f4342273a6df: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:24.185817 15993 catalog_manager.cc:4841] TS b57d5c8da10b4c4ca970f4342273a6df (127.13.238.130:37509): tablet 28993c0cc3c34c36b1679dcc59e299fa (table test-table [id=a56923caf1124eb9ac99f48b8a2d9eb7]) successfully deleted
I20260929 03:15:24.186635 16567 ts_tablet_manager.cc:1918] T 28993c0cc3c34c36b1679dcc59e299fa P 8877d9fe57d64228a82a9fbc4f37a4cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:15:24.186786 16567 log.cc:1196] T 28993c0cc3c34c36b1679dcc59e299fa P 8877d9fe57d64228a82a9fbc4f37a4cb: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/28993c0cc3c34c36b1679dcc59e299fa
I20260929 03:15:24.186898 16423 tablet_service.cc:1467] Processing CreateTablet for tablet c2bb0a40f1884b1ba865537d32d5b1d9 (DEFAULT_TABLE table=test-table [id=ccdba4a7748b4a1c9e6a7524d1193b45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:24.187163 16423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2bb0a40f1884b1ba865537d32d5b1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:24.187183 16567 ts_tablet_manager.cc:1939] T 28993c0cc3c34c36b1679dcc59e299fa P 8877d9fe57d64228a82a9fbc4f37a4cb: Deleting consensus metadata
I20260929 03:15:24.188700 15991 catalog_manager.cc:4841] TS 8877d9fe57d64228a82a9fbc4f37a4cb (127.13.238.131:38703): tablet 28993c0cc3c34c36b1679dcc59e299fa (table test-table [id=a56923caf1124eb9ac99f48b8a2d9eb7]) successfully deleted
I20260929 03:15:24.189138 16514 tablet_bootstrap.cc:492] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb: Bootstrap starting.
I20260929 03:15:24.189924 16514 tablet_bootstrap.cc:654] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:24.190896 16512 tablet_bootstrap.cc:492] T c2bb0a40f1884b1ba865537d32d5b1d9 P b57d5c8da10b4c4ca970f4342273a6df: No bootstrap required, opened a new log
I20260929 03:15:24.190999 16512 ts_tablet_manager.cc:1397] T c2bb0a40f1884b1ba865537d32d5b1d9 P b57d5c8da10b4c4ca970f4342273a6df: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260929 03:15:24.191471 16512 raft_consensus.cc:348] T c2bb0a40f1884b1ba865537d32d5b1d9 P b57d5c8da10b4c4ca970f4342273a6df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:24.191587 16512 raft_consensus.cc:374] T c2bb0a40f1884b1ba865537d32d5b1d9 P b57d5c8da10b4c4ca970f4342273a6df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:24.191603 16514 tablet_bootstrap.cc:492] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb: No bootstrap required, opened a new log
I20260929 03:15:24.191617 16512 raft_consensus.cc:729] T c2bb0a40f1884b1ba865537d32d5b1d9 P b57d5c8da10b4c4ca970f4342273a6df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57d5c8da10b4c4ca970f4342273a6df, State: Initialized, Role: FOLLOWER
I20260929 03:15:24.191674 16514 ts_tablet_manager.cc:1397] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:15:24.191741 16512 consensus_queue.cc:260] T c2bb0a40f1884b1ba865537d32d5b1d9 P b57d5c8da10b4c4ca970f4342273a6df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:24.192008 16512 ts_tablet_manager.cc:1428] T c2bb0a40f1884b1ba865537d32d5b1d9 P b57d5c8da10b4c4ca970f4342273a6df: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:24.192034 16514 raft_consensus.cc:348] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:24.192130 16514 raft_consensus.cc:374] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:24.192158 16514 raft_consensus.cc:729] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8877d9fe57d64228a82a9fbc4f37a4cb, State: Initialized, Role: FOLLOWER
I20260929 03:15:24.192276 16514 consensus_queue.cc:260] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:24.192373 16565 ts_tablet_manager.cc:1918] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:15:24.192463 16565 log.cc:1196] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-0/wal/wals/28993c0cc3c34c36b1679dcc59e299fa
I20260929 03:15:24.192505 16514 ts_tablet_manager.cc:1428] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:24.193231 16513 tablet_bootstrap.cc:492] T c2bb0a40f1884b1ba865537d32d5b1d9 P 10c9b446b29f4e8f966a0c34278bc522: No bootstrap required, opened a new log
I20260929 03:15:24.193333 16513 ts_tablet_manager.cc:1397] T c2bb0a40f1884b1ba865537d32d5b1d9 P 10c9b446b29f4e8f966a0c34278bc522: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260929 03:15:24.193797 16513 raft_consensus.cc:348] T c2bb0a40f1884b1ba865537d32d5b1d9 P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:24.193907 16513 raft_consensus.cc:374] T c2bb0a40f1884b1ba865537d32d5b1d9 P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:24.193938 16513 raft_consensus.cc:729] T c2bb0a40f1884b1ba865537d32d5b1d9 P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c9b446b29f4e8f966a0c34278bc522, State: Initialized, Role: FOLLOWER
I20260929 03:15:24.194072 16513 consensus_queue.cc:260] T c2bb0a40f1884b1ba865537d32d5b1d9 P 10c9b446b29f4e8f966a0c34278bc522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:24.194351 16513 ts_tablet_manager.cc:1428] T c2bb0a40f1884b1ba865537d32d5b1d9 P 10c9b446b29f4e8f966a0c34278bc522: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:24.197538 16565 ts_tablet_manager.cc:1939] T 28993c0cc3c34c36b1679dcc59e299fa P 10c9b446b29f4e8f966a0c34278bc522: Deleting consensus metadata
I20260929 03:15:24.198776 15990 catalog_manager.cc:4841] TS 10c9b446b29f4e8f966a0c34278bc522 (127.13.238.129:34855): tablet 28993c0cc3c34c36b1679dcc59e299fa (table test-table [id=a56923caf1124eb9ac99f48b8a2d9eb7]) successfully deleted
I20260929 03:15:24.440785 16537 raft_consensus.cc:482] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:15:24.440934 16537 raft_consensus.cc:504] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:24.441453 16537 leader_election.cc:288] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10c9b446b29f4e8f966a0c34278bc522 (127.13.238.129:34855), b57d5c8da10b4c4ca970f4342273a6df (127.13.238.130:37509)
I20260929 03:15:24.441884 16163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2bb0a40f1884b1ba865537d32d5b1d9" candidate_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c9b446b29f4e8f966a0c34278bc522" is_pre_election: true
I20260929 03:15:24.441934 16302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2bb0a40f1884b1ba865537d32d5b1d9" candidate_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b57d5c8da10b4c4ca970f4342273a6df" is_pre_election: true
I20260929 03:15:24.442025 16163 raft_consensus.cc:2454] T c2bb0a40f1884b1ba865537d32d5b1d9 P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8877d9fe57d64228a82a9fbc4f37a4cb in term 0.
I20260929 03:15:24.442097 16302 raft_consensus.cc:2454] T c2bb0a40f1884b1ba865537d32d5b1d9 P b57d5c8da10b4c4ca970f4342273a6df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8877d9fe57d64228a82a9fbc4f37a4cb in term 0.
I20260929 03:15:24.442366 16374 leader_election.cc:302] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10c9b446b29f4e8f966a0c34278bc522, 8877d9fe57d64228a82a9fbc4f37a4cb; no voters:
I20260929 03:15:24.442581 16537 raft_consensus.cc:2781] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:15:24.442642 16537 raft_consensus.cc:482] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:15:24.442669 16537 raft_consensus.cc:3037] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:24.443755 16537 raft_consensus.cc:504] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:24.444128 16537 leader_election.cc:288] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb [CANDIDATE]: Term 1 election: Requested vote from peers 10c9b446b29f4e8f966a0c34278bc522 (127.13.238.129:34855), b57d5c8da10b4c4ca970f4342273a6df (127.13.238.130:37509)
I20260929 03:15:24.444551 16163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2bb0a40f1884b1ba865537d32d5b1d9" candidate_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c9b446b29f4e8f966a0c34278bc522"
I20260929 03:15:24.444551 16302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2bb0a40f1884b1ba865537d32d5b1d9" candidate_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b57d5c8da10b4c4ca970f4342273a6df"
I20260929 03:15:24.444694 16163 raft_consensus.cc:3037] T c2bb0a40f1884b1ba865537d32d5b1d9 P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:24.444694 16302 raft_consensus.cc:3037] T c2bb0a40f1884b1ba865537d32d5b1d9 P b57d5c8da10b4c4ca970f4342273a6df [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:24.445955 16302 raft_consensus.cc:2454] T c2bb0a40f1884b1ba865537d32d5b1d9 P b57d5c8da10b4c4ca970f4342273a6df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8877d9fe57d64228a82a9fbc4f37a4cb in term 1.
I20260929 03:15:24.445955 16163 raft_consensus.cc:2454] T c2bb0a40f1884b1ba865537d32d5b1d9 P 10c9b446b29f4e8f966a0c34278bc522 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8877d9fe57d64228a82a9fbc4f37a4cb in term 1.
I20260929 03:15:24.446295 16374 leader_election.cc:302] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 10c9b446b29f4e8f966a0c34278bc522, 8877d9fe57d64228a82a9fbc4f37a4cb, b57d5c8da10b4c4ca970f4342273a6df; no voters:
I20260929 03:15:24.446473 16537 raft_consensus.cc:2781] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:24.446570 16537 raft_consensus.cc:686] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb [term 1 LEADER]: Becoming Leader. State: Replica: 8877d9fe57d64228a82a9fbc4f37a4cb, State: Running, Role: LEADER
I20260929 03:15:24.446715 16537 consensus_queue.cc:237] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:24.448606 16005 catalog_manager.cc:5495] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8877d9fe57d64228a82a9fbc4f37a4cb (127.13.238.131). New cstate: current_term: 1 leader_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } health_report { overall_health: HEALTHY } } }
I20260929 03:15:24.498032 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:24.493831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33924 (local address 127.13.238.131:38703)
0929 03:15:24.493924 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:24.493928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:24.494121 (+ 193us) server_negotiation.cc:407] Connection header received
0929 03:15:24.494166 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:24.494169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:24.494254 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:24.494327 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:24.494666 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.495459 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.496833 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.497146 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.497239 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:24.497551 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:24.497640 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:24.497711 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:24.497850 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 03:15:24.497909 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:15:24.508096 16561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:24.504271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:38703 (local address 127.13.238.131:40079)
0929 03:15:24.504471 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:24.504487 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:24.504581 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:24.504791 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:24.504794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:24.504800 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:15:24.504911 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:24.504917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.505847 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.505850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:24.506763 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:24.506774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.507341 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.507345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:24.507445 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:24.507478 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:15:24.507767 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:15:24.507774 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:15:24.507842 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 03:15:24.507928 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":20}
I20260929 03:15:24.508096 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:24.504338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:40079 (local address 127.13.238.131:38703)
0929 03:15:24.504446 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:24.504450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:24.504539 (+ 89us) server_negotiation.cc:407] Connection header received
0929 03:15:24.504678 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:24.504681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:24.504724 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:24.504784 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:24.505001 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.505728 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.506888 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:24.507209 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:24.507305 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:24.507607 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:24.507681 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:24.507744 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:24.507883 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 03:15:24.507930 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":36}
I20260929 03:15:24.513634 16302 raft_consensus.cc:1261] T c2bb0a40f1884b1ba865537d32d5b1d9 P b57d5c8da10b4c4ca970f4342273a6df [term 1 FOLLOWER]: Refusing update from remote peer 8877d9fe57d64228a82a9fbc4f37a4cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:24.514035 16163 raft_consensus.cc:1261] T c2bb0a40f1884b1ba865537d32d5b1d9 P 10c9b446b29f4e8f966a0c34278bc522 [term 1 FOLLOWER]: Refusing update from remote peer 8877d9fe57d64228a82a9fbc4f37a4cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:24.514006 16537 consensus_queue.cc:1035] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:24.514398 16559 consensus_queue.cc:1035] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 03:15:24.535894 16542 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 03:15:26.092705 16006 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:26.092855 16006 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:26.095372 16006 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc: Sending DeleteTablet for 3 replicas of tablet c2bb0a40f1884b1ba865537d32d5b1d9
I20260929 03:15:26.096011 16143 tablet_service.cc:1514] Processing DeleteTablet for tablet c2bb0a40f1884b1ba865537d32d5b1d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:26 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:37008
I20260929 03:15:26.096037 16423 tablet_service.cc:1514] Processing DeleteTablet for tablet c2bb0a40f1884b1ba865537d32d5b1d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:26 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:33906
I20260929 03:15:26.096642 16596 tablet_replica.cc:330] T c2bb0a40f1884b1ba865537d32d5b1d9 P 10c9b446b29f4e8f966a0c34278bc522: stopping tablet replica
I20260929 03:15:26.096840 16596 raft_consensus.cc:2229] T c2bb0a40f1884b1ba865537d32d5b1d9 P 10c9b446b29f4e8f966a0c34278bc522 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:15:26.097018 16596 raft_consensus.cc:2258] T c2bb0a40f1884b1ba865537d32d5b1d9 P 10c9b446b29f4e8f966a0c34278bc522 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:26.097398 16595 tablet_replica.cc:330] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb: stopping tablet replica
I20260929 03:15:26.097586 16595 raft_consensus.cc:2229] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:15:26.097604 16596 ts_tablet_manager.cc:1905] T c2bb0a40f1884b1ba865537d32d5b1d9 P 10c9b446b29f4e8f966a0c34278bc522: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:26.097988 16595 raft_consensus.cc:2258] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:26.098620 16595 ts_tablet_manager.cc:1905] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:26.099967 16596 ts_tablet_manager.cc:1918] T c2bb0a40f1884b1ba865537d32d5b1d9 P 10c9b446b29f4e8f966a0c34278bc522: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:15:26.100075 16596 log.cc:1196] T c2bb0a40f1884b1ba865537d32d5b1d9 P 10c9b446b29f4e8f966a0c34278bc522: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-0/wal/wals/c2bb0a40f1884b1ba865537d32d5b1d9
I20260929 03:15:26.100083 16283 tablet_service.cc:1514] Processing DeleteTablet for tablet c2bb0a40f1884b1ba865537d32d5b1d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:26 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:42486
I20260929 03:15:26.100343 16596 ts_tablet_manager.cc:1939] T c2bb0a40f1884b1ba865537d32d5b1d9 P 10c9b446b29f4e8f966a0c34278bc522: Deleting consensus metadata
I20260929 03:15:26.101016 15990 catalog_manager.cc:4841] TS 10c9b446b29f4e8f966a0c34278bc522 (127.13.238.129:34855): tablet c2bb0a40f1884b1ba865537d32d5b1d9 (table test-table [id=ccdba4a7748b4a1c9e6a7524d1193b45]) successfully deleted
I20260929 03:15:26.101274 16595 ts_tablet_manager.cc:1918] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:15:26.101353 16595 log.cc:1196] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/c2bb0a40f1884b1ba865537d32d5b1d9
I20260929 03:15:26.101632 16595 ts_tablet_manager.cc:1939] T c2bb0a40f1884b1ba865537d32d5b1d9 P 8877d9fe57d64228a82a9fbc4f37a4cb: Deleting consensus metadata
I20260929 03:15:26.102272 16598 tablet_replica.cc:330] T c2bb0a40f1884b1ba865537d32d5b1d9 P b57d5c8da10b4c4ca970f4342273a6df: stopping tablet replica
I20260929 03:15:26.102303 15991 catalog_manager.cc:4841] TS 8877d9fe57d64228a82a9fbc4f37a4cb (127.13.238.131:38703): tablet c2bb0a40f1884b1ba865537d32d5b1d9 (table test-table [id=ccdba4a7748b4a1c9e6a7524d1193b45]) successfully deleted
I20260929 03:15:26.102406 16598 raft_consensus.cc:2229] T c2bb0a40f1884b1ba865537d32d5b1d9 P b57d5c8da10b4c4ca970f4342273a6df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:15:26.102522 16598 raft_consensus.cc:2258] T c2bb0a40f1884b1ba865537d32d5b1d9 P b57d5c8da10b4c4ca970f4342273a6df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:26.103077 16598 ts_tablet_manager.cc:1905] T c2bb0a40f1884b1ba865537d32d5b1d9 P b57d5c8da10b4c4ca970f4342273a6df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:26.105409 16598 ts_tablet_manager.cc:1918] T c2bb0a40f1884b1ba865537d32d5b1d9 P b57d5c8da10b4c4ca970f4342273a6df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:15:26.105479 16598 log.cc:1196] T c2bb0a40f1884b1ba865537d32d5b1d9 P b57d5c8da10b4c4ca970f4342273a6df: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-1/wal/wals/c2bb0a40f1884b1ba865537d32d5b1d9
I20260929 03:15:26.105683 16598 ts_tablet_manager.cc:1939] T c2bb0a40f1884b1ba865537d32d5b1d9 P b57d5c8da10b4c4ca970f4342273a6df: Deleting consensus metadata
I20260929 03:15:26.106310 15993 catalog_manager.cc:4841] TS b57d5c8da10b4c4ca970f4342273a6df (127.13.238.130:37509): tablet c2bb0a40f1884b1ba865537d32d5b1d9 (table test-table [id=ccdba4a7748b4a1c9e6a7524d1193b45]) successfully deleted
Sep 29 03:15:26 dist-test-slave-nbjg krb5kdc[15961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651726, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:15:26 dist-test-slave-nbjg krb5kdc[15961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651726, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:26.124610 16605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:26.116016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53692 (local address 127.13.238.190:45251)
0929 03:15:26.116277 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:26.116288 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:26.116345 (+ 57us) server_negotiation.cc:407] Connection header received
0929 03:15:26.116488 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:26.116492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:26.116542 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:26.116620 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:26.117336 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:26.117911 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:26.118851 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:26.119098 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:26.119194 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:26.121553 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:26.121556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:26.121559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:26.121590 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:26.122953 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:26.123381 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:26.123384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:26.123386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:26.123469 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:26.123724 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:26.123727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:26.123729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:26.123922 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:26.124056 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:26.124302 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 03:15:26.124435 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":84,"threads_started":1}
I20260929 03:15:26.127210 16006 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:15:26.127650 16006 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:15:26.132215 16143 tablet_service.cc:1467] Processing CreateTablet for tablet 0667d6c8f96d420cbe2c3b8bf4a5e8dd (DEFAULT_TABLE table=test-table [id=cec5f47c18f240bcb1006b06b2796f38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:26.132443 16423 tablet_service.cc:1467] Processing CreateTablet for tablet 0667d6c8f96d420cbe2c3b8bf4a5e8dd (DEFAULT_TABLE table=test-table [id=cec5f47c18f240bcb1006b06b2796f38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:26.132526 16143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0667d6c8f96d420cbe2c3b8bf4a5e8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:26.132443 16283 tablet_service.cc:1467] Processing CreateTablet for tablet 0667d6c8f96d420cbe2c3b8bf4a5e8dd (DEFAULT_TABLE table=test-table [id=cec5f47c18f240bcb1006b06b2796f38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:26.132728 16283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0667d6c8f96d420cbe2c3b8bf4a5e8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:26.132728 16423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0667d6c8f96d420cbe2c3b8bf4a5e8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:26.134668 16606 tablet_bootstrap.cc:492] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522: Bootstrap starting.
I20260929 03:15:26.135185 16607 tablet_bootstrap.cc:492] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P b57d5c8da10b4c4ca970f4342273a6df: Bootstrap starting.
I20260929 03:15:26.135334 16608 tablet_bootstrap.cc:492] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 8877d9fe57d64228a82a9fbc4f37a4cb: Bootstrap starting.
I20260929 03:15:26.135674 16606 tablet_bootstrap.cc:654] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:26.136229 16608 tablet_bootstrap.cc:654] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 8877d9fe57d64228a82a9fbc4f37a4cb: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:26.136410 16607 tablet_bootstrap.cc:654] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P b57d5c8da10b4c4ca970f4342273a6df: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:26.136924 16606 tablet_bootstrap.cc:492] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522: No bootstrap required, opened a new log
I20260929 03:15:26.137013 16606 ts_tablet_manager.cc:1397] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:15:26.137275 16608 tablet_bootstrap.cc:492] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 8877d9fe57d64228a82a9fbc4f37a4cb: No bootstrap required, opened a new log
I20260929 03:15:26.137351 16608 ts_tablet_manager.cc:1397] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 8877d9fe57d64228a82a9fbc4f37a4cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:15:26.137437 16606 raft_consensus.cc:348] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:26.137535 16607 tablet_bootstrap.cc:492] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P b57d5c8da10b4c4ca970f4342273a6df: No bootstrap required, opened a new log
I20260929 03:15:26.137571 16606 raft_consensus.cc:374] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:26.137604 16606 raft_consensus.cc:729] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c9b446b29f4e8f966a0c34278bc522, State: Initialized, Role: FOLLOWER
I20260929 03:15:26.137606 16607 ts_tablet_manager.cc:1397] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P b57d5c8da10b4c4ca970f4342273a6df: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:15:26.137784 16606 consensus_queue.cc:260] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:26.137768 16608 raft_consensus.cc:348] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:26.137882 16608 raft_consensus.cc:374] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:26.137948 16608 raft_consensus.cc:729] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8877d9fe57d64228a82a9fbc4f37a4cb, State: Initialized, Role: FOLLOWER
I20260929 03:15:26.138103 16607 raft_consensus.cc:348] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P b57d5c8da10b4c4ca970f4342273a6df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:26.138108 16608 consensus_queue.cc:260] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 8877d9fe57d64228a82a9fbc4f37a4cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:26.138262 16607 raft_consensus.cc:374] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P b57d5c8da10b4c4ca970f4342273a6df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:26.138309 16607 raft_consensus.cc:729] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P b57d5c8da10b4c4ca970f4342273a6df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57d5c8da10b4c4ca970f4342273a6df, State: Initialized, Role: FOLLOWER
I20260929 03:15:26.138458 16607 consensus_queue.cc:260] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P b57d5c8da10b4c4ca970f4342273a6df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:26.138727 16606 ts_tablet_manager.cc:1428] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:15:26.138797 16607 ts_tablet_manager.cc:1428] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P b57d5c8da10b4c4ca970f4342273a6df: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 03:15:26.139703 16608 ts_tablet_manager.cc:1428] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 8877d9fe57d64228a82a9fbc4f37a4cb: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 03:15:26.166288 16612 raft_consensus.cc:482] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:15:26.166419 16612 raft_consensus.cc:504] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:26.166775 16612 leader_election.cc:288] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b57d5c8da10b4c4ca970f4342273a6df (127.13.238.130:37509), 8877d9fe57d64228a82a9fbc4f37a4cb (127.13.238.131:38703)
I20260929 03:15:26.167173 16443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0667d6c8f96d420cbe2c3b8bf4a5e8dd" candidate_uuid: "10c9b446b29f4e8f966a0c34278bc522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" is_pre_election: true
I20260929 03:15:26.167312 16443 raft_consensus.cc:2454] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c9b446b29f4e8f966a0c34278bc522 in term 0.
I20260929 03:15:26.167627 16095 leader_election.cc:302] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10c9b446b29f4e8f966a0c34278bc522, 8877d9fe57d64228a82a9fbc4f37a4cb; no voters:
I20260929 03:15:26.167589 16302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0667d6c8f96d420cbe2c3b8bf4a5e8dd" candidate_uuid: "10c9b446b29f4e8f966a0c34278bc522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b57d5c8da10b4c4ca970f4342273a6df" is_pre_election: true
I20260929 03:15:26.167743 16302 raft_consensus.cc:2454] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P b57d5c8da10b4c4ca970f4342273a6df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c9b446b29f4e8f966a0c34278bc522 in term 0.
I20260929 03:15:26.167936 16612 raft_consensus.cc:2781] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:15:26.167990 16612 raft_consensus.cc:482] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:15:26.168025 16612 raft_consensus.cc:3037] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:26.168996 16612 raft_consensus.cc:504] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:26.169301 16612 leader_election.cc:288] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522 [CANDIDATE]: Term 1 election: Requested vote from peers b57d5c8da10b4c4ca970f4342273a6df (127.13.238.130:37509), 8877d9fe57d64228a82a9fbc4f37a4cb (127.13.238.131:38703)
I20260929 03:15:26.169642 16443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0667d6c8f96d420cbe2c3b8bf4a5e8dd" candidate_uuid: "10c9b446b29f4e8f966a0c34278bc522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb"
I20260929 03:15:26.169634 16302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0667d6c8f96d420cbe2c3b8bf4a5e8dd" candidate_uuid: "10c9b446b29f4e8f966a0c34278bc522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b57d5c8da10b4c4ca970f4342273a6df"
I20260929 03:15:26.169754 16302 raft_consensus.cc:3037] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P b57d5c8da10b4c4ca970f4342273a6df [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:26.169754 16443 raft_consensus.cc:3037] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:26.170845 16443 raft_consensus.cc:2454] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 8877d9fe57d64228a82a9fbc4f37a4cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c9b446b29f4e8f966a0c34278bc522 in term 1.
I20260929 03:15:26.170845 16302 raft_consensus.cc:2454] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P b57d5c8da10b4c4ca970f4342273a6df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c9b446b29f4e8f966a0c34278bc522 in term 1.
I20260929 03:15:26.171118 16095 leader_election.cc:302] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10c9b446b29f4e8f966a0c34278bc522, 8877d9fe57d64228a82a9fbc4f37a4cb; no voters:
I20260929 03:15:26.171257 16612 raft_consensus.cc:2781] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:26.171489 16612 raft_consensus.cc:686] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522 [term 1 LEADER]: Becoming Leader. State: Replica: 10c9b446b29f4e8f966a0c34278bc522, State: Running, Role: LEADER
I20260929 03:15:26.171619 16612 consensus_queue.cc:237] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:26.173146 16006 catalog_manager.cc:5495] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10c9b446b29f4e8f966a0c34278bc522 (127.13.238.129). New cstate: current_term: 1 leader_uuid: "10c9b446b29f4e8f966a0c34278bc522" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20260929 03:15:26.196983 16617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:26.193831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46346 (local address 127.13.238.129:34855)
0929 03:15:26.194100 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:26.194110 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:26.194137 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:15:26.194174 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:26.194176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:26.194243 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:26.194336 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:26.194525 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:26.195122 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:26.196034 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:26.196254 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:26.196346 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:26.196602 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:26.196671 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:26.196724 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:26.196786 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 03:15:26.196828 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":113,"threads_started":1}
I20260929 03:15:26.202550 16443 raft_consensus.cc:1261] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 8877d9fe57d64228a82a9fbc4f37a4cb [term 1 FOLLOWER]: Refusing update from remote peer 10c9b446b29f4e8f966a0c34278bc522: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:26.202636 16302 raft_consensus.cc:1261] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P b57d5c8da10b4c4ca970f4342273a6df [term 1 FOLLOWER]: Refusing update from remote peer 10c9b446b29f4e8f966a0c34278bc522: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:26.202915 16614 consensus_queue.cc:1035] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:26.203191 16612 consensus_queue.cc:1035] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:26.229764 16006 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:26.229918 16006 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:26.231907 16006 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc: Sending DeleteTablet for 3 replicas of tablet 0667d6c8f96d420cbe2c3b8bf4a5e8dd
I20260929 03:15:26.232410 16143 tablet_service.cc:1514] Processing DeleteTablet for tablet 0667d6c8f96d420cbe2c3b8bf4a5e8dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:26 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:37008
I20260929 03:15:26.232504 16283 tablet_service.cc:1514] Processing DeleteTablet for tablet 0667d6c8f96d420cbe2c3b8bf4a5e8dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:26 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:42486
I20260929 03:15:26.232517 16423 tablet_service.cc:1514] Processing DeleteTablet for tablet 0667d6c8f96d420cbe2c3b8bf4a5e8dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:26 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:33906
I20260929 03:15:26.232592 16596 tablet_replica.cc:330] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522: stopping tablet replica
I20260929 03:15:26.232663 16595 tablet_replica.cc:330] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 8877d9fe57d64228a82a9fbc4f37a4cb: stopping tablet replica
I20260929 03:15:26.232707 16598 tablet_replica.cc:330] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P b57d5c8da10b4c4ca970f4342273a6df: stopping tablet replica
I20260929 03:15:26.232707 16596 raft_consensus.cc:2229] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:15:26.232767 16595 raft_consensus.cc:2229] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 8877d9fe57d64228a82a9fbc4f37a4cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:15:26.232802 16598 raft_consensus.cc:2229] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P b57d5c8da10b4c4ca970f4342273a6df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:15:26.232854 16596 raft_consensus.cc:2258] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:26.232863 16595 raft_consensus.cc:2258] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 8877d9fe57d64228a82a9fbc4f37a4cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:26.232906 16598 raft_consensus.cc:2258] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P b57d5c8da10b4c4ca970f4342273a6df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:26.233418 16596 ts_tablet_manager.cc:1905] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:26.233531 16595 ts_tablet_manager.cc:1905] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 8877d9fe57d64228a82a9fbc4f37a4cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:26.233982 16598 ts_tablet_manager.cc:1905] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P b57d5c8da10b4c4ca970f4342273a6df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:26.235635 16596 ts_tablet_manager.cc:1918] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:15:26.235695 16596 log.cc:1196] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-0/wal/wals/0667d6c8f96d420cbe2c3b8bf4a5e8dd
I20260929 03:15:26.235901 16596 ts_tablet_manager.cc:1939] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 10c9b446b29f4e8f966a0c34278bc522: Deleting consensus metadata
I20260929 03:15:26.236115 16595 ts_tablet_manager.cc:1918] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 8877d9fe57d64228a82a9fbc4f37a4cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:15:26.236183 16595 log.cc:1196] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 8877d9fe57d64228a82a9fbc4f37a4cb: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/0667d6c8f96d420cbe2c3b8bf4a5e8dd
I20260929 03:15:26.236331 16598 ts_tablet_manager.cc:1918] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P b57d5c8da10b4c4ca970f4342273a6df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:15:26.236389 16598 log.cc:1196] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P b57d5c8da10b4c4ca970f4342273a6df: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-1/wal/wals/0667d6c8f96d420cbe2c3b8bf4a5e8dd
I20260929 03:15:26.236402 16595 ts_tablet_manager.cc:1939] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P 8877d9fe57d64228a82a9fbc4f37a4cb: Deleting consensus metadata
I20260929 03:15:26.236588 16598 ts_tablet_manager.cc:1939] T 0667d6c8f96d420cbe2c3b8bf4a5e8dd P b57d5c8da10b4c4ca970f4342273a6df: Deleting consensus metadata
I20260929 03:15:26.236724 15990 catalog_manager.cc:4841] TS 10c9b446b29f4e8f966a0c34278bc522 (127.13.238.129:34855): tablet 0667d6c8f96d420cbe2c3b8bf4a5e8dd (table test-table [id=cec5f47c18f240bcb1006b06b2796f38]) successfully deleted
I20260929 03:15:26.237104 15991 catalog_manager.cc:4841] TS 8877d9fe57d64228a82a9fbc4f37a4cb (127.13.238.131:38703): tablet 0667d6c8f96d420cbe2c3b8bf4a5e8dd (table test-table [id=cec5f47c18f240bcb1006b06b2796f38]) successfully deleted
I20260929 03:15:26.237156 15993 catalog_manager.cc:4841] TS b57d5c8da10b4c4ca970f4342273a6df (127.13.238.130:37509): tablet 0667d6c8f96d420cbe2c3b8bf4a5e8dd (table test-table [id=cec5f47c18f240bcb1006b06b2796f38]) successfully deleted
I20260929 03:15:26.237375 16006 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:15:26.237777 16006 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:15:26.242378 16143 tablet_service.cc:1467] Processing CreateTablet for tablet f315b9b96191447bbc7aabd6361273bc (DEFAULT_TABLE table=test-table [id=6eec306e884f48aa87bc9a7ac6190e48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:26.242416 16283 tablet_service.cc:1467] Processing CreateTablet for tablet f315b9b96191447bbc7aabd6361273bc (DEFAULT_TABLE table=test-table [id=6eec306e884f48aa87bc9a7ac6190e48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:26.242635 16143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f315b9b96191447bbc7aabd6361273bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:26.242659 16283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f315b9b96191447bbc7aabd6361273bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:26.242619 16423 tablet_service.cc:1467] Processing CreateTablet for tablet f315b9b96191447bbc7aabd6361273bc (DEFAULT_TABLE table=test-table [id=6eec306e884f48aa87bc9a7ac6190e48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:26.242898 16423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f315b9b96191447bbc7aabd6361273bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:26.244813 16606 tablet_bootstrap.cc:492] T f315b9b96191447bbc7aabd6361273bc P 10c9b446b29f4e8f966a0c34278bc522: Bootstrap starting.
I20260929 03:15:26.244894 16607 tablet_bootstrap.cc:492] T f315b9b96191447bbc7aabd6361273bc P b57d5c8da10b4c4ca970f4342273a6df: Bootstrap starting.
I20260929 03:15:26.245643 16606 tablet_bootstrap.cc:654] T f315b9b96191447bbc7aabd6361273bc P 10c9b446b29f4e8f966a0c34278bc522: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:26.245684 16607 tablet_bootstrap.cc:654] T f315b9b96191447bbc7aabd6361273bc P b57d5c8da10b4c4ca970f4342273a6df: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:26.246495 16608 tablet_bootstrap.cc:492] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb: Bootstrap starting.
I20260929 03:15:26.246790 16607 tablet_bootstrap.cc:492] T f315b9b96191447bbc7aabd6361273bc P b57d5c8da10b4c4ca970f4342273a6df: No bootstrap required, opened a new log
I20260929 03:15:26.246877 16607 ts_tablet_manager.cc:1397] T f315b9b96191447bbc7aabd6361273bc P b57d5c8da10b4c4ca970f4342273a6df: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 03:15:26.247352 16608 tablet_bootstrap.cc:654] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:26.247300 16607 raft_consensus.cc:348] T f315b9b96191447bbc7aabd6361273bc P b57d5c8da10b4c4ca970f4342273a6df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:26.247406 16607 raft_consensus.cc:374] T f315b9b96191447bbc7aabd6361273bc P b57d5c8da10b4c4ca970f4342273a6df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:26.247450 16607 raft_consensus.cc:729] T f315b9b96191447bbc7aabd6361273bc P b57d5c8da10b4c4ca970f4342273a6df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57d5c8da10b4c4ca970f4342273a6df, State: Initialized, Role: FOLLOWER
I20260929 03:15:26.247581 16607 consensus_queue.cc:260] T f315b9b96191447bbc7aabd6361273bc P b57d5c8da10b4c4ca970f4342273a6df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:26.247648 16606 tablet_bootstrap.cc:492] T f315b9b96191447bbc7aabd6361273bc P 10c9b446b29f4e8f966a0c34278bc522: No bootstrap required, opened a new log
I20260929 03:15:26.247725 16606 ts_tablet_manager.cc:1397] T f315b9b96191447bbc7aabd6361273bc P 10c9b446b29f4e8f966a0c34278bc522: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:15:26.247840 16607 ts_tablet_manager.cc:1428] T f315b9b96191447bbc7aabd6361273bc P b57d5c8da10b4c4ca970f4342273a6df: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 03:15:26.248114 16606 raft_consensus.cc:348] T f315b9b96191447bbc7aabd6361273bc P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:26.248214 16606 raft_consensus.cc:374] T f315b9b96191447bbc7aabd6361273bc P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:26.248242 16606 raft_consensus.cc:729] T f315b9b96191447bbc7aabd6361273bc P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c9b446b29f4e8f966a0c34278bc522, State: Initialized, Role: FOLLOWER
I20260929 03:15:26.248365 16606 consensus_queue.cc:260] T f315b9b96191447bbc7aabd6361273bc P 10c9b446b29f4e8f966a0c34278bc522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:26.248616 16606 ts_tablet_manager.cc:1428] T f315b9b96191447bbc7aabd6361273bc P 10c9b446b29f4e8f966a0c34278bc522: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:26.248752 16608 tablet_bootstrap.cc:492] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb: No bootstrap required, opened a new log
I20260929 03:15:26.248826 16608 ts_tablet_manager.cc:1397] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 03:15:26.249210 16608 raft_consensus.cc:348] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:26.249331 16608 raft_consensus.cc:374] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:26.249373 16608 raft_consensus.cc:729] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8877d9fe57d64228a82a9fbc4f37a4cb, State: Initialized, Role: FOLLOWER
I20260929 03:15:26.249488 16608 consensus_queue.cc:260] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:26.249758 16608 ts_tablet_manager.cc:1428] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 03:15:26.303615 16578 raft_consensus.cc:482] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:15:26.303839 16578 raft_consensus.cc:504] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:26.304306 16578 leader_election.cc:288] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10c9b446b29f4e8f966a0c34278bc522 (127.13.238.129:34855), b57d5c8da10b4c4ca970f4342273a6df (127.13.238.130:37509)
I20260929 03:15:26.304811 16162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f315b9b96191447bbc7aabd6361273bc" candidate_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c9b446b29f4e8f966a0c34278bc522" is_pre_election: true
I20260929 03:15:26.304821 16302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f315b9b96191447bbc7aabd6361273bc" candidate_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b57d5c8da10b4c4ca970f4342273a6df" is_pre_election: true
I20260929 03:15:26.304950 16162 raft_consensus.cc:2454] T f315b9b96191447bbc7aabd6361273bc P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8877d9fe57d64228a82a9fbc4f37a4cb in term 0.
I20260929 03:15:26.304958 16302 raft_consensus.cc:2454] T f315b9b96191447bbc7aabd6361273bc P b57d5c8da10b4c4ca970f4342273a6df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8877d9fe57d64228a82a9fbc4f37a4cb in term 0.
I20260929 03:15:26.305287 16377 leader_election.cc:302] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8877d9fe57d64228a82a9fbc4f37a4cb, b57d5c8da10b4c4ca970f4342273a6df; no voters:
I20260929 03:15:26.305455 16578 raft_consensus.cc:2781] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:15:26.305524 16578 raft_consensus.cc:482] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:15:26.305564 16578 raft_consensus.cc:3037] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:26.306623 16578 raft_consensus.cc:504] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:26.306952 16578 leader_election.cc:288] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb [CANDIDATE]: Term 1 election: Requested vote from peers 10c9b446b29f4e8f966a0c34278bc522 (127.13.238.129:34855), b57d5c8da10b4c4ca970f4342273a6df (127.13.238.130:37509)
I20260929 03:15:26.307317 16162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f315b9b96191447bbc7aabd6361273bc" candidate_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c9b446b29f4e8f966a0c34278bc522"
I20260929 03:15:26.307334 16302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f315b9b96191447bbc7aabd6361273bc" candidate_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b57d5c8da10b4c4ca970f4342273a6df"
I20260929 03:15:26.307451 16162 raft_consensus.cc:3037] T f315b9b96191447bbc7aabd6361273bc P 10c9b446b29f4e8f966a0c34278bc522 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:26.307462 16302 raft_consensus.cc:3037] T f315b9b96191447bbc7aabd6361273bc P b57d5c8da10b4c4ca970f4342273a6df [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:26.308333 16302 raft_consensus.cc:2454] T f315b9b96191447bbc7aabd6361273bc P b57d5c8da10b4c4ca970f4342273a6df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8877d9fe57d64228a82a9fbc4f37a4cb in term 1.
I20260929 03:15:26.308503 16162 raft_consensus.cc:2454] T f315b9b96191447bbc7aabd6361273bc P 10c9b446b29f4e8f966a0c34278bc522 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8877d9fe57d64228a82a9fbc4f37a4cb in term 1.
I20260929 03:15:26.308650 16377 leader_election.cc:302] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8877d9fe57d64228a82a9fbc4f37a4cb, b57d5c8da10b4c4ca970f4342273a6df; no voters:
I20260929 03:15:26.308825 16578 raft_consensus.cc:2781] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:26.308915 16578 raft_consensus.cc:686] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb [term 1 LEADER]: Becoming Leader. State: Replica: 8877d9fe57d64228a82a9fbc4f37a4cb, State: Running, Role: LEADER
I20260929 03:15:26.309087 16578 consensus_queue.cc:237] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } }
I20260929 03:15:26.310663 16004 catalog_manager.cc:5495] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8877d9fe57d64228a82a9fbc4f37a4cb (127.13.238.131). New cstate: current_term: 1 leader_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8877d9fe57d64228a82a9fbc4f37a4cb" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38703 } health_report { overall_health: HEALTHY } } }
I20260929 03:15:26.337674 16626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:26.334553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36044 (local address 127.13.238.131:38703)
0929 03:15:26.334829 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:26.334838 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:26.334851 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:15:26.334890 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:26.334893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:26.334936 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:26.335004 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:26.335245 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:26.335848 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:26.336669 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:26.336892 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:26.336977 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:26.337233 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:26.337304 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:26.337369 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:26.337466 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 03:15:26.337519 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":109,"threads_started":1}
I20260929 03:15:26.343397 16302 raft_consensus.cc:1261] T f315b9b96191447bbc7aabd6361273bc P b57d5c8da10b4c4ca970f4342273a6df [term 1 FOLLOWER]: Refusing update from remote peer 8877d9fe57d64228a82a9fbc4f37a4cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:26.343397 16162 raft_consensus.cc:1261] T f315b9b96191447bbc7aabd6361273bc P 10c9b446b29f4e8f966a0c34278bc522 [term 1 FOLLOWER]: Refusing update from remote peer 8877d9fe57d64228a82a9fbc4f37a4cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:26.343722 16578 consensus_queue.cc:1035] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b57d5c8da10b4c4ca970f4342273a6df" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:26.343874 16537 consensus_queue.cc:1035] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "10c9b446b29f4e8f966a0c34278bc522" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 34855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:26.345884 16576 mvcc.cc:205] Tried to move back new op lower bound from 7334509471099826176 to 7334509470963605504. Current Snapshot: MvccSnapshot[applied={T|T < 7334509471099826176}]
W20260929 03:15:26.356523 16576 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 03:15:27.904240 16004 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:27.904390 16004 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:27.906642 16004 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c14f64fa6dcb41e482ec0c1c203b41bc: Sending DeleteTablet for 3 replicas of tablet f315b9b96191447bbc7aabd6361273bc
I20260929 03:15:27.907146 16143 tablet_service.cc:1514] Processing DeleteTablet for tablet f315b9b96191447bbc7aabd6361273bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:27 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:37008
I20260929 03:15:27.907192 16283 tablet_service.cc:1514] Processing DeleteTablet for tablet f315b9b96191447bbc7aabd6361273bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:27 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:42486
I20260929 03:15:27.907280 16423 tablet_service.cc:1514] Processing DeleteTablet for tablet f315b9b96191447bbc7aabd6361273bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:27 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:33906
I20260929 03:15:27.907523 16651 tablet_replica.cc:330] T f315b9b96191447bbc7aabd6361273bc P b57d5c8da10b4c4ca970f4342273a6df: stopping tablet replica
I20260929 03:15:27.907572 16652 tablet_replica.cc:330] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb: stopping tablet replica
I20260929 03:15:27.907649 16651 raft_consensus.cc:2229] T f315b9b96191447bbc7aabd6361273bc P b57d5c8da10b4c4ca970f4342273a6df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:15:27.907721 16652 raft_consensus.cc:2229] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:15:27.907761 16651 raft_consensus.cc:2258] T f315b9b96191447bbc7aabd6361273bc P b57d5c8da10b4c4ca970f4342273a6df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:27.907783 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 16078
I20260929 03:15:27.908948 16652 raft_consensus.cc:2258] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:27.909476 16652 ts_tablet_manager.cc:1905] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:27.910753 16651 ts_tablet_manager.cc:1905] T f315b9b96191447bbc7aabd6361273bc P b57d5c8da10b4c4ca970f4342273a6df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:27.911680 16652 ts_tablet_manager.cc:1918] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:15:27.911746 16652 log.cc:1196] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/f315b9b96191447bbc7aabd6361273bc
I20260929 03:15:27.912004 16652 ts_tablet_manager.cc:1939] T f315b9b96191447bbc7aabd6361273bc P 8877d9fe57d64228a82a9fbc4f37a4cb: Deleting consensus metadata
I20260929 03:15:27.912590 15991 catalog_manager.cc:4841] TS 8877d9fe57d64228a82a9fbc4f37a4cb (127.13.238.131:38703): tablet f315b9b96191447bbc7aabd6361273bc (table test-table [id=6eec306e884f48aa87bc9a7ac6190e48]) successfully deleted
I20260929 03:15:27.913154 16651 ts_tablet_manager.cc:1918] T f315b9b96191447bbc7aabd6361273bc P b57d5c8da10b4c4ca970f4342273a6df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:15:27.913225 16651 log.cc:1196] T f315b9b96191447bbc7aabd6361273bc P b57d5c8da10b4c4ca970f4342273a6df: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790651713532531-14266-0/minicluster-data/ts-1/wal/wals/f315b9b96191447bbc7aabd6361273bc
I20260929 03:15:27.913450 16651 ts_tablet_manager.cc:1939] T f315b9b96191447bbc7aabd6361273bc P b57d5c8da10b4c4ca970f4342273a6df: Deleting consensus metadata
I20260929 03:15:27.914033 15993 catalog_manager.cc:4841] TS b57d5c8da10b4c4ca970f4342273a6df (127.13.238.130:37509): tablet f315b9b96191447bbc7aabd6361273bc (table test-table [id=6eec306e884f48aa87bc9a7ac6190e48]) successfully deleted
W20260929 03:15:27.915467 15990 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.238.129:34855 (error 108)
I20260929 03:15:27.915629 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 16218
I20260929 03:15:27.917855 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:27.917450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:34855 (local address 127.0.0.1:46352)
0929 03:15:27.917689 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:27.917771 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.129:34855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":92,"threads_started":1}
W20260929 03:15:27.917991 15990 catalog_manager.cc:4584] TS 10c9b446b29f4e8f966a0c34278bc522 (127.13.238.129:34855): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f315b9b96191447bbc7aabd6361273bc: Network error: Client connection negotiation failed: client connection to 127.13.238.129:34855: connect: Connection refused (error 111)
I20260929 03:15:27.920960 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 16358
I20260929 03:15:27.926558 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 15974
2026-09-29T03:15:27Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6647 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:15:27 dist-test-slave-nbjg krb5kdc[16657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:15:27 dist-test-slave-nbjg krb5kdc[16657](info): set up 2 sockets
Sep 29 03:15:27 dist-test-slave-nbjg krb5kdc[16657](info): commencing operation
krb5kdc: starting...
W20260929 03:15:28.882155 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:15:28 dist-test-slave-nbjg krb5kdc[16657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:15:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:15:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:15:28.993517 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:37381
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:37381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:29.102880 16670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:29.103220 16670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:29.103279 16670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:29.107313 16670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:29.107370 16670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:29.107398 16670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:29.107419 16670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:29.107447 16670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:15:29.112970 16670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:37381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:37381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:29.114125 16670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:29.115234 16670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:15:29.124290 16670 server_base.cc:1034] running on GCE node
W20260929 03:15:29.124250 16676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:29.124258 16678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:29.124248 16675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:29.125114 16670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:29.126211 16670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:29.127389 16670 hybrid_clock.cc:648] HybridClock initialized: now 1790651729127369 us; error 29 us; skew 500 ppm
Sep 29 03:15:29 dist-test-slave-nbjg krb5kdc[16657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651729, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:29.130030 16670 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:15:29.130939 16670 webserver.cc:460] Webserver started at http://127.13.238.190:36703/ using document root <none> and password file <none>
I20260929 03:15:29.131487 16670 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:29.131553 16670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:29.131793 16670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:29.133916 16670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "c3e2f2d20b664d1aa740f40b3c6479ab"
format_stamp: "Formatted at 2026-09-29 03:15:29 on dist-test-slave-nbjg"
server_key: "26349637e3cfe7bc34434535fad9d89e"
server_key_iv: "59f67ea48f47f57e6de25b9d08246b35"
server_key_version: "encryptionkey@0"
I20260929 03:15:29.134508 16670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "c3e2f2d20b664d1aa740f40b3c6479ab"
format_stamp: "Formatted at 2026-09-29 03:15:29 on dist-test-slave-nbjg"
server_key: "26349637e3cfe7bc34434535fad9d89e"
server_key_iv: "59f67ea48f47f57e6de25b9d08246b35"
server_key_version: "encryptionkey@0"
I20260929 03:15:29.138578 16670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 03:15:29.141017 16685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:29.142311 16670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:29.142424 16670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "c3e2f2d20b664d1aa740f40b3c6479ab"
format_stamp: "Formatted at 2026-09-29 03:15:29 on dist-test-slave-nbjg"
server_key: "26349637e3cfe7bc34434535fad9d89e"
server_key_iv: "59f67ea48f47f57e6de25b9d08246b35"
server_key_version: "encryptionkey@0"
I20260929 03:15:29.142519 16670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:29.160243 16670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:29.163249 16670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:29.163432 16670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:29.170176 16670 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:37381
I20260929 03:15:29.170234 16737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:37381 every 8 connection(s)
I20260929 03:15:29.171319 16670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:15:29.174849 16738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:29.179569 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 16670
I20260929 03:15:29.179679 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:15:29.180006 14266 external_mini_cluster.cc:1428] Setting key 0c1ebc1dc9e5cd961e696f1fd0f3f2b4
I20260929 03:15:29.181041 16738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3e2f2d20b664d1aa740f40b3c6479ab: Bootstrap starting.
I20260929 03:15:29.183490 16738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3e2f2d20b664d1aa740f40b3c6479ab: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:29.184197 16738 log.cc:824] T 00000000000000000000000000000000 P c3e2f2d20b664d1aa740f40b3c6479ab: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:29.186045 16738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3e2f2d20b664d1aa740f40b3c6479ab: No bootstrap required, opened a new log
Sep 29 03:15:29 dist-test-slave-nbjg krb5kdc[16657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:29.190021 16738 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3e2f2d20b664d1aa740f40b3c6479ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3e2f2d20b664d1aa740f40b3c6479ab" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37381 } }
I20260929 03:15:29.190362 16738 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3e2f2d20b664d1aa740f40b3c6479ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:29.190510 16738 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3e2f2d20b664d1aa740f40b3c6479ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e2f2d20b664d1aa740f40b3c6479ab, State: Initialized, Role: FOLLOWER
I20260929 03:15:29.191018 16738 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3e2f2d20b664d1aa740f40b3c6479ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3e2f2d20b664d1aa740f40b3c6479ab" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37381 } }
I20260929 03:15:29.191169 16738 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3e2f2d20b664d1aa740f40b3c6479ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:15:29.191253 16738 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3e2f2d20b664d1aa740f40b3c6479ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:15:29.191390 16738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3e2f2d20b664d1aa740f40b3c6479ab [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:29.192461 16738 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3e2f2d20b664d1aa740f40b3c6479ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3e2f2d20b664d1aa740f40b3c6479ab" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37381 } }
I20260929 03:15:29.192776 16738 leader_election.cc:302] T 00000000000000000000000000000000 P c3e2f2d20b664d1aa740f40b3c6479ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3e2f2d20b664d1aa740f40b3c6479ab; no voters:
I20260929 03:15:29.193096 16738 leader_election.cc:288] T 00000000000000000000000000000000 P c3e2f2d20b664d1aa740f40b3c6479ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:15:29.193171 16743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3e2f2d20b664d1aa740f40b3c6479ab [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:29.193572 16743 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3e2f2d20b664d1aa740f40b3c6479ab [term 1 LEADER]: Becoming Leader. State: Replica: c3e2f2d20b664d1aa740f40b3c6479ab, State: Running, Role: LEADER
I20260929 03:15:29.193935 16743 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3e2f2d20b664d1aa740f40b3c6479ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3e2f2d20b664d1aa740f40b3c6479ab" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37381 } }
I20260929 03:15:29.194416 16738 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3e2f2d20b664d1aa740f40b3c6479ab [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:29.194723 16741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:29.181566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54964 (local address 127.13.238.190:37381)
0929 03:15:29.182037 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:29.182052 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:29.182088 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:15:29.182759 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:29.182777 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:29.183127 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:29.183434 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:29.184201 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:29.185222 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:29.186060 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:29.186601 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:29.186745 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:29.189385 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:29.189390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:29.189404 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:29.189431 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:29.191475 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:29.191967 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:29.191972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:29.191976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:29.192088 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:29.192375 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:29.192377 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:29.192378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:29.192790 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:29.192949 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:29.193314 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 03:15:29.193635 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":146,"threads_started":1}
I20260929 03:15:29.195574 16744 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3e2f2d20b664d1aa740f40b3c6479ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3e2f2d20b664d1aa740f40b3c6479ab. Latest consensus state: current_term: 1 leader_uuid: "c3e2f2d20b664d1aa740f40b3c6479ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3e2f2d20b664d1aa740f40b3c6479ab" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37381 } } }
I20260929 03:15:29.195688 16744 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3e2f2d20b664d1aa740f40b3c6479ab [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:29.196072 16745 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3e2f2d20b664d1aa740f40b3c6479ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3e2f2d20b664d1aa740f40b3c6479ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3e2f2d20b664d1aa740f40b3c6479ab" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37381 } } }
I20260929 03:15:29.196202 16745 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3e2f2d20b664d1aa740f40b3c6479ab [sys.catalog]: This master's current role is: LEADER
W20260929 03:15:29.199172 16758 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c3e2f2d20b664d1aa740f40b3c6479ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:15:29.199236 16758 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:15:29.199331 16752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:15:29.200089 16752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:15:29.206463 16752 catalog_manager.cc:1347] Generated new cluster ID: 93d6a53358b242daa5cbd4aa9c31af68
I20260929 03:15:29.206528 16752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:15:29.216738 16752 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:15:29.217818 16752 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:15:29.223239 16752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3e2f2d20b664d1aa740f40b3c6479ab: Generated new TSK 0
I20260929 03:15:29.223982 16752 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
I20260929 03:15:29.251418 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:0
--local_ip_for_outbound_sockets=127.13.238.129
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:37381
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:29.377040 16764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:29.377278 16764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:29.377318 16764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:29.381230 16764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:29.381295 16764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:29.381359 16764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:15:29.386567 16764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:37381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:29.387626 16764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:29.388638 16764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:15:29.398696 16764 server_base.cc:1034] running on GCE node
W20260929 03:15:29.398630 16772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:29.398634 16770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:29.398630 16769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:29.399258 16764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:29.399958 16764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:29.401130 16764 hybrid_clock.cc:648] HybridClock initialized: now 1790651729401098 us; error 45 us; skew 500 ppm
Sep 29 03:15:29 dist-test-slave-nbjg krb5kdc[16657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651729, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:29.403957 16764 init.cc:377] Logged in from keytab as kudu/127.13.238.129@KRBTEST.COM (short username kudu)
I20260929 03:15:29.404917 16764 webserver.cc:460] Webserver started at http://127.13.238.129:35203/ using document root <none> and password file <none>
I20260929 03:15:29.405508 16764 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:29.405577 16764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:29.405773 16764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:29.407986 16764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/data/instance:
uuid: "cdf191b7ce3e4a4694c8256a5d63c2f6"
format_stamp: "Formatted at 2026-09-29 03:15:29 on dist-test-slave-nbjg"
server_key: "320c9e703ab2a7755634d89e2c365f98"
server_key_iv: "50c5b5a85905680bfca0f7786129b141"
server_key_version: "encryptionkey@0"
I20260929 03:15:29.408560 16764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance:
uuid: "cdf191b7ce3e4a4694c8256a5d63c2f6"
format_stamp: "Formatted at 2026-09-29 03:15:29 on dist-test-slave-nbjg"
server_key: "320c9e703ab2a7755634d89e2c365f98"
server_key_iv: "50c5b5a85905680bfca0f7786129b141"
server_key_version: "encryptionkey@0"
I20260929 03:15:29.412773 16764 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260929 03:15:29.415546 16779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:29.416870 16764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:15:29.416985 16764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "cdf191b7ce3e4a4694c8256a5d63c2f6"
format_stamp: "Formatted at 2026-09-29 03:15:29 on dist-test-slave-nbjg"
server_key: "320c9e703ab2a7755634d89e2c365f98"
server_key_iv: "50c5b5a85905680bfca0f7786129b141"
server_key_version: "encryptionkey@0"
I20260929 03:15:29.417102 16764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:29.439666 16764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:29.442799 16764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:29.442981 16764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:29.443593 16764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:29.444459 16764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:29.444506 16764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:29.444553 16764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:29.444583 16764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:29.453112 16764 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:37241
I20260929 03:15:29.453162 16892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:37241 every 8 connection(s)
I20260929 03:15:29.454071 16764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:15:29.458750 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 16764
I20260929 03:15:29.458842 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance
I20260929 03:15:29.459076 14266 external_mini_cluster.cc:1428] Setting key 1826b45a10988d5f7c1ef2b4061c75b2
Sep 29 03:15:29 dist-test-slave-nbjg krb5kdc[16657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651729, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:29.468323 16741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:29.455853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:52773 (local address 127.13.238.190:37381)
0929 03:15:29.455985 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:29.455988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:29.456837 (+ 849us) server_negotiation.cc:407] Connection header received
0929 03:15:29.458273 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:29.458276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:29.458323 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:29.458394 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:29.459693 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:29.460289 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:29.461473 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:29.461789 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:29.461896 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:29.464540 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:29.464544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:29.464547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:29.464576 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:29.466053 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:29.466557 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:29.466561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:29.466563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:29.466659 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:29.466974 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:29.466978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:29.466980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:29.467210 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:29.467313 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:29.467897 (+ 584us) server_negotiation.cc:299] Negotiation successful
0929 03:15:29.468082 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.13.238.130@KRBTEST.COM; defaulting to no policy
I20260929 03:15:29.469242 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:29.456237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:37381 (local address 127.13.238.129:52773)
0929 03:15:29.456683 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:29.456725 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:29.458033 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:29.458595 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:29.458603 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:29.458983 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:29.459554 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:29.459576 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:29.460411 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:29.460414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:29.461343 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:29.461353 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:29.461968 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:29.461973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:29.462169 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:29.462773 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:29.462793 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:29.464369 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:29.466161 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:29.466166 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:29.466179 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:29.466463 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:29.466749 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:29.466751 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:29.466752 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:29.466881 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:29.467297 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:29.467302 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:29.467581 (+ 279us) client_negotiation.cc:769] Sending connection context
0929 03:15:29.467778 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 03:15:29.468147 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":132,"threads_started":1}
I20260929 03:15:29.470552 16893 heartbeater.cc:347] Connected to a master server at 127.13.238.190:37381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.130@KRBTEST.COM" created.
I20260929 03:15:29.470875 16893 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:29.471424 16893 heartbeater.cc:511] Master 127.13.238.190:37381 requested a full tablet report, sending...
I20260929 03:15:29.473188 16702 ts_manager.cc:194] Registered new tserver with Master: cdf191b7ce3e4a4694c8256a5d63c2f6 (127.13.238.129:37241)
I20260929 03:15:29.474490 16702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.129@KRBTEST.COM'} at 127.13.238.129:52773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
I20260929 03:15:29.485103 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:0
--local_ip_for_outbound_sockets=127.13.238.130
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:37381
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:29.598820 16898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:29.599086 16898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:29.599149 16898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:29.603245 16898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:29.603310 16898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:29.603389 16898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:15:29.608762 16898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:37381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:29.609838 16898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:29.610944 16898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:29.620222 16904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:29.620216 16903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:29.620306 16898 server_base.cc:1034] running on GCE node
W20260929 03:15:29.620213 16906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:29.620855 16898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:29.621483 16898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:29.622634 16898 hybrid_clock.cc:648] HybridClock initialized: now 1790651729622616 us; error 31 us; skew 500 ppm
Sep 29 03:15:29 dist-test-slave-nbjg krb5kdc[16657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651729, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:29.629408 16898 init.cc:377] Logged in from keytab as kudu/127.13.238.130@KRBTEST.COM (short username kudu)
I20260929 03:15:29.630366 16898 webserver.cc:460] Webserver started at http://127.13.238.130:41139/ using document root <none> and password file <none>
I20260929 03:15:29.630919 16898 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:29.630983 16898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:29.631182 16898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:29.633299 16898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/data/instance:
uuid: "65a6c5ce87f444388997138ec69a42e0"
format_stamp: "Formatted at 2026-09-29 03:15:29 on dist-test-slave-nbjg"
server_key: "8999545fccf45c8f3e1dfd3b33e25e65"
server_key_iv: "97c48498e7866f4fd6375baa254612ed"
server_key_version: "encryptionkey@0"
I20260929 03:15:29.633831 16898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance:
uuid: "65a6c5ce87f444388997138ec69a42e0"
format_stamp: "Formatted at 2026-09-29 03:15:29 on dist-test-slave-nbjg"
server_key: "8999545fccf45c8f3e1dfd3b33e25e65"
server_key_iv: "97c48498e7866f4fd6375baa254612ed"
server_key_version: "encryptionkey@0"
I20260929 03:15:29.637493 16898 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 03:15:29.639971 16913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:29.641322 16898 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:15:29.641440 16898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "65a6c5ce87f444388997138ec69a42e0"
format_stamp: "Formatted at 2026-09-29 03:15:29 on dist-test-slave-nbjg"
server_key: "8999545fccf45c8f3e1dfd3b33e25e65"
server_key_iv: "97c48498e7866f4fd6375baa254612ed"
server_key_version: "encryptionkey@0"
I20260929 03:15:29.641537 16898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:29.653789 16898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:29.656528 16898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:29.656698 16898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:29.657272 16898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:29.658143 16898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:29.658190 16898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:29.658272 16898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:29.658296 16898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:29.666527 16898 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:38981
I20260929 03:15:29.666553 17026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:38981 every 8 connection(s)
I20260929 03:15:29.667371 16898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:15:29.670953 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 16898
I20260929 03:15:29.671052 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance
I20260929 03:15:29.671291 14266 external_mini_cluster.cc:1428] Setting key a3b37e75e6de76a51437d71119c8744f
Sep 29 03:15:29 dist-test-slave-nbjg krb5kdc[16657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651729, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.238.131@KRBTEST.COM; defaulting to no policy
I20260929 03:15:29.681664 16741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:29.669237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:59839 (local address 127.13.238.190:37381)
0929 03:15:29.669373 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:29.669377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:29.670167 (+ 790us) server_negotiation.cc:407] Connection header received
0929 03:15:29.671526 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:29.671529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:29.671587 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:29.671657 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:29.673016 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:29.673527 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:29.675130 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:29.675361 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:29.675447 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:29.678137 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:29.678142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:29.678145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:29.678175 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:29.679699 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:29.680189 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:29.680193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:29.680195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:29.680282 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:29.680588 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:29.680591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:29.680593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:29.680826 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:29.680943 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:29.681418 (+ 475us) server_negotiation.cc:299] Negotiation successful
0929 03:15:29.681540 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:15:29.682626 17029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:29.669585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:37381 (local address 127.13.238.130:59839)
0929 03:15:29.670007 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:29.670049 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:29.671298 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:29.671839 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:29.671847 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:29.672333 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:29.672869 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:29.672883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:29.673644 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:29.673648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:29.674987 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:29.675000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:29.675509 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:29.675513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:29.675700 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:29.676305 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:29.676325 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:29.677949 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:29.679808 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:29.679813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:29.679828 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:29.680089 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:29.680366 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:29.680368 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:29.680369 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:29.680501 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:29.680919 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:29.680924 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:29.681167 (+ 243us) client_negotiation.cc:769] Sending connection context
0929 03:15:29.681344 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 03:15:29.681629 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.131@KRBTEST.COM" created.
I20260929 03:15:29.683905 17027 heartbeater.cc:347] Connected to a master server at 127.13.238.190:37381
I20260929 03:15:29.684243 17027 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:29.684772 17027 heartbeater.cc:511] Master 127.13.238.190:37381 requested a full tablet report, sending...
I20260929 03:15:29.685866 16702 ts_manager.cc:194] Registered new tserver with Master: 65a6c5ce87f444388997138ec69a42e0 (127.13.238.130:38981)
I20260929 03:15:29.686461 16702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.130@KRBTEST.COM'} at 127.13.238.130:59839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
I20260929 03:15:29.697175 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:0
--local_ip_for_outbound_sockets=127.13.238.131
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:37381
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:29.809048 17032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:29.809348 17032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:29.809408 17032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:29.813656 17032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:29.813719 17032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:29.813808 17032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:15:29.819132 17032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:37381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:29.820202 17032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:29.821262 17032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:29.830888 17037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:29.830955 17038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:29.830947 17032 server_base.cc:1034] running on GCE node
W20260929 03:15:29.830864 17040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:29.831552 17032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:29.832371 17032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:29.833563 17032 hybrid_clock.cc:648] HybridClock initialized: now 1790651729833540 us; error 38 us; skew 500 ppm
Sep 29 03:15:29 dist-test-slave-nbjg krb5kdc[16657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651729, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:29.836457 17032 init.cc:377] Logged in from keytab as kudu/127.13.238.131@KRBTEST.COM (short username kudu)
I20260929 03:15:29.837345 17032 webserver.cc:460] Webserver started at http://127.13.238.131:38115/ using document root <none> and password file <none>
I20260929 03:15:29.837963 17032 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:29.838032 17032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:29.838256 17032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:29.840294 17032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/data/instance:
uuid: "a9f077e51d754ea4b4a66b123d1446e7"
format_stamp: "Formatted at 2026-09-29 03:15:29 on dist-test-slave-nbjg"
server_key: "811203efae8b3c812bb2af9631c381c3"
server_key_iv: "701a08ed4f6bfe7abc13afb540b4304a"
server_key_version: "encryptionkey@0"
I20260929 03:15:29.840837 17032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance:
uuid: "a9f077e51d754ea4b4a66b123d1446e7"
format_stamp: "Formatted at 2026-09-29 03:15:29 on dist-test-slave-nbjg"
server_key: "811203efae8b3c812bb2af9631c381c3"
server_key_iv: "701a08ed4f6bfe7abc13afb540b4304a"
server_key_version: "encryptionkey@0"
I20260929 03:15:29.844560 17032 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.002s
I20260929 03:15:29.847280 17047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:29.848534 17032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:29.848645 17032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "a9f077e51d754ea4b4a66b123d1446e7"
format_stamp: "Formatted at 2026-09-29 03:15:29 on dist-test-slave-nbjg"
server_key: "811203efae8b3c812bb2af9631c381c3"
server_key_iv: "701a08ed4f6bfe7abc13afb540b4304a"
server_key_version: "encryptionkey@0"
I20260929 03:15:29.848743 17032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:29.884729 17032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:29.887753 17032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:29.887936 17032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:29.888546 17032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:29.889386 17032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:29.889433 17032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:29.889487 17032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:29.889516 17032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:29.898283 17032 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:35529
I20260929 03:15:29.898303 17160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:35529 every 8 connection(s)
I20260929 03:15:29.899222 17032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:15:29.903314 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 17032
I20260929 03:15:29.903422 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance
I20260929 03:15:29.903716 14266 external_mini_cluster.cc:1428] Setting key ab3829c584a116ab019885bc1be9abe9
Sep 29 03:15:29 dist-test-slave-nbjg krb5kdc[16657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651729, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:29.912904 16741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:29.901091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:45491 (local address 127.13.238.190:37381)
0929 03:15:29.901240 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:29.901244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:29.902139 (+ 895us) server_negotiation.cc:407] Connection header received
0929 03:15:29.903569 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:29.903572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:29.903641 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:29.903723 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:29.905169 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:29.905718 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:29.906976 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:29.907192 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:29.907272 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:29.909614 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:29.909617 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:29.909620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:29.909651 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:29.911021 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:29.911462 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:29.911464 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:29.911466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:29.911536 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:29.911845 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:29.911849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:29.911850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:29.912054 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:29.912140 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:29.912647 (+ 507us) server_negotiation.cc:299] Negotiation successful
0929 03:15:29.912761 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 03:15:29.913769 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:29.901502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:37381 (local address 127.13.238.131:45491)
0929 03:15:29.901983 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:29.902029 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:29.903332 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:29.903932 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:29.903940 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:29.904403 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:29.905028 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:29.905047 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:29.905845 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:29.905849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:29.906851 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:29.906862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:29.907313 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:29.907315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:29.907476 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:29.907984 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:29.908001 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:29.909432 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:29.911110 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:29.911115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:29.911130 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:29.911385 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:29.911597 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:29.911599 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:29.911601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:29.911731 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:29.912146 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:29.912151 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:29.912412 (+ 261us) client_negotiation.cc:769] Sending connection context
0929 03:15:29.912589 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 03:15:29.912862 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":124,"threads_started":1}
I20260929 03:15:29.914956 17161 heartbeater.cc:347] Connected to a master server at 127.13.238.190:37381
I20260929 03:15:29.915205 17161 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:29.915709 17161 heartbeater.cc:511] Master 127.13.238.190:37381 requested a full tablet report, sending...
I20260929 03:15:29.916857 16702 ts_manager.cc:194] Registered new tserver with Master: a9f077e51d754ea4b4a66b123d1446e7 (127.13.238.131:35529)
I20260929 03:15:29.917557 16702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.131@KRBTEST.COM'} at 127.13.238.131:45491
I20260929 03:15:29.917701 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 03:15:29.930500 16741 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:29.929121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54972 (local address 127.13.238.190:37381)
0929 03:15:29.929259 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:29.929264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:29.929396 (+ 132us) server_negotiation.cc:407] Connection header received
0929 03:15:29.929511 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:29.929513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:29.929557 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:29.929620 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:29.930353 (+ 733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54972: BlockingRecv error: recv got EOF from 127.0.0.1:54972 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:15:29.931931 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 16764
I20260929 03:15:29.937458 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 16898
I20260929 03:15:29.942026 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 17032
I20260929 03:15:29.946520 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 16670
2026-09-29T03:15:29Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2019 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:15:30 dist-test-slave-nbjg krb5kdc[17173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:15:30 dist-test-slave-nbjg krb5kdc[17173](info): set up 2 sockets
Sep 29 03:15:30 dist-test-slave-nbjg krb5kdc[17173](info): commencing operation
krb5kdc: starting...
W20260929 03:15:30.898233 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:15:30 dist-test-slave-nbjg krb5kdc[17173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:15:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:15:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:15:31.010974 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:35463
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:35463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:31.121430 17186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:31.121716 17186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:31.121758 17186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:31.125905 17186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:31.125976 17186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:31.125995 17186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:31.126011 17186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:31.126025 17186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:15:31.131428 17186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:35463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:35463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:31.132551 17186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:31.133605 17186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:15:31.142005 17186 server_base.cc:1034] running on GCE node
W20260929 03:15:31.142032 17191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:31.142028 17192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:31.142001 17194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:31.142800 17186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:31.143848 17186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:31.145000 17186 hybrid_clock.cc:648] HybridClock initialized: now 1790651731144976 us; error 39 us; skew 500 ppm
Sep 29 03:15:31 dist-test-slave-nbjg krb5kdc[17173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651731, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:31.147671 17186 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:15:31.148533 17186 webserver.cc:460] Webserver started at http://127.13.238.190:37693/ using document root <none> and password file <none>
I20260929 03:15:31.149080 17186 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:31.149143 17186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:31.149340 17186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:31.151374 17186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "aa242d2ad34443dca72e84bde85c88d3"
format_stamp: "Formatted at 2026-09-29 03:15:31 on dist-test-slave-nbjg"
server_key: "415b8db176cba567f7e81986b21c5b43"
server_key_iv: "6ca2402c875290f9dffe404a047af466"
server_key_version: "encryptionkey@0"
I20260929 03:15:31.151988 17186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "aa242d2ad34443dca72e84bde85c88d3"
format_stamp: "Formatted at 2026-09-29 03:15:31 on dist-test-slave-nbjg"
server_key: "415b8db176cba567f7e81986b21c5b43"
server_key_iv: "6ca2402c875290f9dffe404a047af466"
server_key_version: "encryptionkey@0"
I20260929 03:15:31.155615 17186 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:15:31.157976 17201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:31.159202 17186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:31.159315 17186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "aa242d2ad34443dca72e84bde85c88d3"
format_stamp: "Formatted at 2026-09-29 03:15:31 on dist-test-slave-nbjg"
server_key: "415b8db176cba567f7e81986b21c5b43"
server_key_iv: "6ca2402c875290f9dffe404a047af466"
server_key_version: "encryptionkey@0"
I20260929 03:15:31.159407 17186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:31.176133 17186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:31.179437 17186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:31.179637 17186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:31.186600 17186 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:35463
I20260929 03:15:31.186890 17253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:35463 every 8 connection(s)
I20260929 03:15:31.187584 17186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:15:31.190464 17254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:31.196261 17254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa242d2ad34443dca72e84bde85c88d3: Bootstrap starting.
I20260929 03:15:31.196870 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 17186
I20260929 03:15:31.196972 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:15:31.197203 14266 external_mini_cluster.cc:1428] Setting key 6b71a79b5ce18f4dddc233ac98367169
I20260929 03:15:31.199031 17254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa242d2ad34443dca72e84bde85c88d3: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:31.199932 17254 log.cc:824] T 00000000000000000000000000000000 P aa242d2ad34443dca72e84bde85c88d3: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:31.202343 17254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa242d2ad34443dca72e84bde85c88d3: No bootstrap required, opened a new log
I20260929 03:15:31.206895 17254 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa242d2ad34443dca72e84bde85c88d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa242d2ad34443dca72e84bde85c88d3" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 35463 } }
I20260929 03:15:31.207192 17254 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa242d2ad34443dca72e84bde85c88d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:31.207262 17254 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa242d2ad34443dca72e84bde85c88d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa242d2ad34443dca72e84bde85c88d3, State: Initialized, Role: FOLLOWER
I20260929 03:15:31.207700 17254 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa242d2ad34443dca72e84bde85c88d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa242d2ad34443dca72e84bde85c88d3" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 35463 } }
I20260929 03:15:31.207857 17254 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa242d2ad34443dca72e84bde85c88d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:15:31.207921 17254 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa242d2ad34443dca72e84bde85c88d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:15:31.208012 17254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa242d2ad34443dca72e84bde85c88d3 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 03:15:31 dist-test-slave-nbjg krb5kdc[17173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:31.209046 17254 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa242d2ad34443dca72e84bde85c88d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa242d2ad34443dca72e84bde85c88d3" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 35463 } }
I20260929 03:15:31.209415 17254 leader_election.cc:302] T 00000000000000000000000000000000 P aa242d2ad34443dca72e84bde85c88d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa242d2ad34443dca72e84bde85c88d3; no voters:
I20260929 03:15:31.209767 17254 leader_election.cc:288] T 00000000000000000000000000000000 P aa242d2ad34443dca72e84bde85c88d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:15:31.209873 17259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa242d2ad34443dca72e84bde85c88d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:31.210095 17259 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa242d2ad34443dca72e84bde85c88d3 [term 1 LEADER]: Becoming Leader. State: Replica: aa242d2ad34443dca72e84bde85c88d3, State: Running, Role: LEADER
I20260929 03:15:31.210431 17259 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa242d2ad34443dca72e84bde85c88d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa242d2ad34443dca72e84bde85c88d3" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 35463 } }
I20260929 03:15:31.211196 17254 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa242d2ad34443dca72e84bde85c88d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:31.211700 17261 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa242d2ad34443dca72e84bde85c88d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa242d2ad34443dca72e84bde85c88d3. Latest consensus state: current_term: 1 leader_uuid: "aa242d2ad34443dca72e84bde85c88d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa242d2ad34443dca72e84bde85c88d3" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 35463 } } }
I20260929 03:15:31.211819 17261 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa242d2ad34443dca72e84bde85c88d3 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:31.212127 17260 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa242d2ad34443dca72e84bde85c88d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa242d2ad34443dca72e84bde85c88d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa242d2ad34443dca72e84bde85c88d3" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 35463 } } }
I20260929 03:15:31.212224 17260 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa242d2ad34443dca72e84bde85c88d3 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:31.214805 17257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:31.198583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37076 (local address 127.13.238.190:35463)
0929 03:15:31.199036 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:31.199056 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:31.199101 (+ 45us) server_negotiation.cc:407] Connection header received
0929 03:15:31.199966 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:31.199993 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:31.200486 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:31.200869 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:31.202346 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:31.203826 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:31.205088 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:31.205901 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:31.206064 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:31.208868 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:31.208874 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:31.208893 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:31.208926 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:31.210839 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:31.211374 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:31.211380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:31.211386 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:31.211535 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:31.211870 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:31.211873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:31.211875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:31.212450 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:31.212633 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:31.213484 (+ 851us) server_negotiation.cc:299] Negotiation successful
0929 03:15:31.213793 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":124,"threads_started":1}
W20260929 03:15:31.216028 17274 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa242d2ad34443dca72e84bde85c88d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:15:31.216104 17274 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:15:31.216163 17264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:15:31.216986 17264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:15:31.218003 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:15:31.222890 17264 catalog_manager.cc:1347] Generated new cluster ID: a4235e7048334c8b82085aa7993040ea
I20260929 03:15:31.222968 17264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:15:31.237608 17264 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:15:31.238791 17264 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:15:31.239486 17257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:31.233615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37082 (local address 127.13.238.190:35463)
0929 03:15:31.233744 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:31.233748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:31.234039 (+ 291us) server_negotiation.cc:407] Connection header received
0929 03:15:31.234164 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:31.234167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:31.234250 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:31.234338 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:31.234942 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:31.235842 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:31.236913 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:31.237204 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:31.237298 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:31.238337 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:31.238341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:31.238344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:15:31.238966 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:31.239003 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:15:31.239024 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:31.239119 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:31.239244 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 03:15:31.239339 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 03:15:31.244267 17264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa242d2ad34443dca72e84bde85c88d3: Generated new TSK 0
I20260929 03:15:31.245102 17264 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:15:31.254344 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 17186
2026-09-29T03:15:31Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1306 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:15:31 dist-test-slave-nbjg krb5kdc[17287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:15:31 dist-test-slave-nbjg krb5kdc[17287](info): set up 2 sockets
Sep 29 03:15:31 dist-test-slave-nbjg krb5kdc[17287](info): commencing operation
krb5kdc: starting...
W20260929 03:15:32.214123 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:15:32 dist-test-slave-nbjg krb5kdc[17287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:15:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:15:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:15:32.328995 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:45743
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:45743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:32.439893 17300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:32.440155 17300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:32.440217 17300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:32.444219 17300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:32.444271 17300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:32.444296 17300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:32.444319 17300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:32.444346 17300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:15:32.449579 17300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:45743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:45743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:32.450718 17300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:32.451736 17300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:32.459949 17308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:32.459975 17306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:32.459983 17305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:32.460090 17300 server_base.cc:1034] running on GCE node
I20260929 03:15:32.460667 17300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:32.461733 17300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:32.462899 17300 hybrid_clock.cc:648] HybridClock initialized: now 1790651732462873 us; error 34 us; skew 500 ppm
Sep 29 03:15:32 dist-test-slave-nbjg krb5kdc[17287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651732, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:32.465600 17300 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:15:32.466547 17300 webserver.cc:460] Webserver started at http://127.13.238.190:42813/ using document root <none> and password file <none>
I20260929 03:15:32.467082 17300 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:32.467145 17300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:32.467347 17300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:32.469347 17300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "b4ab870c1671472aa2499a54e50e04e8"
format_stamp: "Formatted at 2026-09-29 03:15:32 on dist-test-slave-nbjg"
server_key: "c6289b35bff7e58f794fdf12142a3c3c"
server_key_iv: "71405a66759fd3a0fc04f4f44631687a"
server_key_version: "encryptionkey@0"
I20260929 03:15:32.469895 17300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "b4ab870c1671472aa2499a54e50e04e8"
format_stamp: "Formatted at 2026-09-29 03:15:32 on dist-test-slave-nbjg"
server_key: "c6289b35bff7e58f794fdf12142a3c3c"
server_key_iv: "71405a66759fd3a0fc04f4f44631687a"
server_key_version: "encryptionkey@0"
I20260929 03:15:32.473502 17300 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 03:15:32.475957 17315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:32.477198 17300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 03:15:32.477317 17300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "b4ab870c1671472aa2499a54e50e04e8"
format_stamp: "Formatted at 2026-09-29 03:15:32 on dist-test-slave-nbjg"
server_key: "c6289b35bff7e58f794fdf12142a3c3c"
server_key_iv: "71405a66759fd3a0fc04f4f44631687a"
server_key_version: "encryptionkey@0"
I20260929 03:15:32.477412 17300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:32.494122 17300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:32.496971 17300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:32.497150 17300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:32.503831 17300 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:45743
I20260929 03:15:32.504096 17367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:45743 every 8 connection(s)
I20260929 03:15:32.504779 17300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:15:32.507611 17368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:32.513218 17368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4ab870c1671472aa2499a54e50e04e8: Bootstrap starting.
I20260929 03:15:32.514729 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 17300
I20260929 03:15:32.514848 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:15:32.515115 14266 external_mini_cluster.cc:1428] Setting key ec02b11f95ddcfa55365f5383e001616
I20260929 03:15:32.516130 17368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4ab870c1671472aa2499a54e50e04e8: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:32.516906 17368 log.cc:824] T 00000000000000000000000000000000 P b4ab870c1671472aa2499a54e50e04e8: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:32.519150 17368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4ab870c1671472aa2499a54e50e04e8: No bootstrap required, opened a new log
I20260929 03:15:32.523329 17368 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4ab870c1671472aa2499a54e50e04e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ab870c1671472aa2499a54e50e04e8" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 45743 } }
I20260929 03:15:32.523633 17368 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4ab870c1671472aa2499a54e50e04e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:32.523705 17368 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4ab870c1671472aa2499a54e50e04e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4ab870c1671472aa2499a54e50e04e8, State: Initialized, Role: FOLLOWER
Sep 29 03:15:32 dist-test-slave-nbjg krb5kdc[17287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:32.524187 17368 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4ab870c1671472aa2499a54e50e04e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ab870c1671472aa2499a54e50e04e8" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 45743 } }
I20260929 03:15:32.524338 17368 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4ab870c1671472aa2499a54e50e04e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:15:32.524400 17368 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4ab870c1671472aa2499a54e50e04e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:15:32.524497 17368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4ab870c1671472aa2499a54e50e04e8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:32.525666 17368 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4ab870c1671472aa2499a54e50e04e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ab870c1671472aa2499a54e50e04e8" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 45743 } }
I20260929 03:15:32.526087 17368 leader_election.cc:302] T 00000000000000000000000000000000 P b4ab870c1671472aa2499a54e50e04e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4ab870c1671472aa2499a54e50e04e8; no voters:
I20260929 03:15:32.526468 17368 leader_election.cc:288] T 00000000000000000000000000000000 P b4ab870c1671472aa2499a54e50e04e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:15:32.526575 17373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4ab870c1671472aa2499a54e50e04e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:32.526774 17373 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4ab870c1671472aa2499a54e50e04e8 [term 1 LEADER]: Becoming Leader. State: Replica: b4ab870c1671472aa2499a54e50e04e8, State: Running, Role: LEADER
I20260929 03:15:32.527189 17373 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4ab870c1671472aa2499a54e50e04e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ab870c1671472aa2499a54e50e04e8" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 45743 } }
I20260929 03:15:32.527980 17368 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4ab870c1671472aa2499a54e50e04e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:32.528908 17374 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4ab870c1671472aa2499a54e50e04e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4ab870c1671472aa2499a54e50e04e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ab870c1671472aa2499a54e50e04e8" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 45743 } } }
I20260929 03:15:32.529034 17374 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4ab870c1671472aa2499a54e50e04e8 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:32.529517 17375 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4ab870c1671472aa2499a54e50e04e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4ab870c1671472aa2499a54e50e04e8. Latest consensus state: current_term: 1 leader_uuid: "b4ab870c1671472aa2499a54e50e04e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ab870c1671472aa2499a54e50e04e8" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 45743 } } }
I20260929 03:15:32.529672 17375 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4ab870c1671472aa2499a54e50e04e8 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:32.530352 17382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:15:32.530941 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:32.516313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56390 (local address 127.13.238.190:45743)
0929 03:15:32.516736 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:32.516752 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:32.516789 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:15:32.517545 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:32.517571 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:32.517975 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:32.518298 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:32.519189 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:32.520229 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:32.521451 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:32.521954 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:32.522086 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:32.524823 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:32.524830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:32.524849 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:32.524881 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:32.526823 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:32.527409 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:32.527414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:32.527420 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:32.527573 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:32.527936 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:32.527939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:32.527941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:32.528497 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:32.529306 (+ 809us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:32.529490 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 03:15:32.529819 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":123,"threads_started":1}
I20260929 03:15:32.533299 17382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:15:32.538930 17382 catalog_manager.cc:1347] Generated new cluster ID: f0156fbb1deb45b1b1829ef72a0ac7ea
I20260929 03:15:32.538995 17382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:15:32.561882 17382 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:15:32.563259 17382 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:15:32.582120 17382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4ab870c1671472aa2499a54e50e04e8: Generated new TSK 0
I20260929 03:15:32.582898 17382 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
I20260929 03:15:32.613368 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:0
--local_ip_for_outbound_sockets=127.13.238.129
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:45743
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:32.723718 17394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:32.723996 17394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:32.724041 17394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:32.728071 17394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:32.728139 17394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:32.728209 17394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:15:32.733606 17394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:45743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:32.734769 17394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:32.735821 17394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:15:32.745627 17394 server_base.cc:1034] running on GCE node
W20260929 03:15:32.745525 17399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:32.745541 17400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:32.745527 17402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:32.746296 17394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:32.746930 17394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:32.748106 17394 hybrid_clock.cc:648] HybridClock initialized: now 1790651732748084 us; error 31 us; skew 500 ppm
Sep 29 03:15:32 dist-test-slave-nbjg krb5kdc[17287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651732, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:32.750947 17394 init.cc:377] Logged in from keytab as kudu/127.13.238.129@KRBTEST.COM (short username kudu)
I20260929 03:15:32.751799 17394 webserver.cc:460] Webserver started at http://127.13.238.129:39561/ using document root <none> and password file <none>
I20260929 03:15:32.752344 17394 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:32.752405 17394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:32.752621 17394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:32.754709 17394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/data/instance:
uuid: "478a5416484d4b05bcf8115f6c67aba3"
format_stamp: "Formatted at 2026-09-29 03:15:32 on dist-test-slave-nbjg"
server_key: "b8559822aef4cb8003b5b1683ebda1bb"
server_key_iv: "f1468a41b71799817aff14d846360651"
server_key_version: "encryptionkey@0"
I20260929 03:15:32.755249 17394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance:
uuid: "478a5416484d4b05bcf8115f6c67aba3"
format_stamp: "Formatted at 2026-09-29 03:15:32 on dist-test-slave-nbjg"
server_key: "b8559822aef4cb8003b5b1683ebda1bb"
server_key_iv: "f1468a41b71799817aff14d846360651"
server_key_version: "encryptionkey@0"
I20260929 03:15:32.759009 17394 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 03:15:32.761453 17409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:32.762823 17394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:32.762930 17394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "478a5416484d4b05bcf8115f6c67aba3"
format_stamp: "Formatted at 2026-09-29 03:15:32 on dist-test-slave-nbjg"
server_key: "b8559822aef4cb8003b5b1683ebda1bb"
server_key_iv: "f1468a41b71799817aff14d846360651"
server_key_version: "encryptionkey@0"
I20260929 03:15:32.763049 17394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:32.776707 17394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:32.780166 17394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:32.780370 17394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:32.780980 17394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:32.781875 17394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:32.781926 17394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:32.781975 17394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:32.781996 17394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:32.789969 17394 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:43429
I20260929 03:15:32.790025 17522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:43429 every 8 connection(s)
I20260929 03:15:32.790902 17394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:15:32.799553 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 17394
I20260929 03:15:32.799655 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance
I20260929 03:15:32.799935 14266 external_mini_cluster.cc:1428] Setting key 927fb20884dee1aa299f9b4214978b91
Sep 29 03:15:32 dist-test-slave-nbjg krb5kdc[17287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651732, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:32.804369 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:32.792742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:44001 (local address 127.13.238.190:45743)
0929 03:15:32.792905 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:32.792908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:32.793703 (+ 795us) server_negotiation.cc:407] Connection header received
0929 03:15:32.795096 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:32.795100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:32.795157 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:32.795229 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:32.796409 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:32.796945 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:32.798084 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:32.798364 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:32.798458 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:32.800821 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:32.800825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:32.800827 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:32.800851 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:32.802334 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:32.802938 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:32.802942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:32.802944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:32.803033 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:32.803327 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:32.803330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:32.803331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:32.803535 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:32.803630 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:32.804147 (+ 517us) server_negotiation.cc:299] Negotiation successful
0929 03:15:32.804251 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:15:32.805331 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:32.793099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:45743 (local address 127.13.238.129:44001)
0929 03:15:32.793554 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:32.793598 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:32.794884 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:32.795423 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:32.795431 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:32.795790 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:32.796262 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:32.796275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:32.797062 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:32.797065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:32.797966 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:32.797975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:32.798511 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:32.798515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:32.798677 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:32.799197 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:32.799214 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:32.800664 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:32.802456 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:32.802462 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:32.802479 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:32.802833 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:32.803117 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:32.803119 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:32.803121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:32.803255 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:32.803639 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:32.803644 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:32.803928 (+ 284us) client_negotiation.cc:769] Sending connection context
0929 03:15:32.804089 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 03:15:32.804341 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":103,"threads_started":1}
I20260929 03:15:32.806602 17523 heartbeater.cc:347] Connected to a master server at 127.13.238.190:45743
I20260929 03:15:32.806828 17523 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:32.807305 17523 heartbeater.cc:511] Master 127.13.238.190:45743 requested a full tablet report, sending...
I20260929 03:15:32.808818 17332 ts_manager.cc:194] Registered new tserver with Master: 478a5416484d4b05bcf8115f6c67aba3 (127.13.238.129:43429)
WARNING: no policy specified for kudu/127.13.238.130@KRBTEST.COM; defaulting to no policy
I20260929 03:15:32.809973 17332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.129@KRBTEST.COM'} at 127.13.238.129:44001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
I20260929 03:15:32.825672 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:0
--local_ip_for_outbound_sockets=127.13.238.130
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:45743
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:32.935943 17528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:32.936223 17528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:32.936268 17528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:32.940390 17528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:32.940474 17528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:32.940557 17528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:15:32.945977 17528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:45743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:32.947082 17528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:32.948213 17528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:32.958505 17534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:32.958509 17533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:32.958630 17528 server_base.cc:1034] running on GCE node
W20260929 03:15:32.958505 17536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:32.959208 17528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:32.959906 17528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:32.961068 17528 hybrid_clock.cc:648] HybridClock initialized: now 1790651732961040 us; error 41 us; skew 500 ppm
Sep 29 03:15:32 dist-test-slave-nbjg krb5kdc[17287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651732, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:32.963835 17528 init.cc:377] Logged in from keytab as kudu/127.13.238.130@KRBTEST.COM (short username kudu)
I20260929 03:15:32.964766 17528 webserver.cc:460] Webserver started at http://127.13.238.130:42603/ using document root <none> and password file <none>
I20260929 03:15:32.965341 17528 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:32.965415 17528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:32.965637 17528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:32.968047 17528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/data/instance:
uuid: "ea9af098f2354ba890608a55419c6424"
format_stamp: "Formatted at 2026-09-29 03:15:32 on dist-test-slave-nbjg"
server_key: "3fdca0fb11ee02e429299d01cc33f07c"
server_key_iv: "47f2f621487ca0a192307edfee022e08"
server_key_version: "encryptionkey@0"
I20260929 03:15:32.968621 17528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance:
uuid: "ea9af098f2354ba890608a55419c6424"
format_stamp: "Formatted at 2026-09-29 03:15:32 on dist-test-slave-nbjg"
server_key: "3fdca0fb11ee02e429299d01cc33f07c"
server_key_iv: "47f2f621487ca0a192307edfee022e08"
server_key_version: "encryptionkey@0"
I20260929 03:15:32.972491 17528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:15:32.975092 17543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:32.976321 17528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:32.976440 17528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "ea9af098f2354ba890608a55419c6424"
format_stamp: "Formatted at 2026-09-29 03:15:32 on dist-test-slave-nbjg"
server_key: "3fdca0fb11ee02e429299d01cc33f07c"
server_key_iv: "47f2f621487ca0a192307edfee022e08"
server_key_version: "encryptionkey@0"
I20260929 03:15:32.976547 17528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:32.993530 17528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:32.996531 17528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:32.996711 17528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:32.997330 17528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:32.998250 17528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:32.998296 17528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:32.998342 17528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:32.998369 17528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:33.006861 17528 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:33707
I20260929 03:15:33.006886 17656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:33707 every 8 connection(s)
I20260929 03:15:33.007737 17528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:15:33.011739 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 17528
I20260929 03:15:33.011862 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance
I20260929 03:15:33.012188 14266 external_mini_cluster.cc:1428] Setting key 15f68ad13bc428ce0303b72be619da56
Sep 29 03:15:33 dist-test-slave-nbjg krb5kdc[17287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651732, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.238.131@KRBTEST.COM; defaulting to no policy
I20260929 03:15:33.022068 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:33.009491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:59947 (local address 127.13.238.190:45743)
0929 03:15:33.009657 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:33.009663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:33.010538 (+ 875us) server_negotiation.cc:407] Connection header received
0929 03:15:33.012225 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:33.012228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:33.012289 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:33.012363 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:33.013799 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:33.014365 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:33.015814 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:33.016038 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:33.016126 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:33.018641 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:33.018645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:33.018647 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:33.018673 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:33.020058 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:33.020560 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:33.020564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:33.020565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:33.020644 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:33.020974 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:33.020978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:33.020980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:33.021169 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:33.021256 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:33.021804 (+ 548us) server_negotiation.cc:299] Negotiation successful
0929 03:15:33.021938 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 03:15:33.022981 17659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:33.009905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:45743 (local address 127.13.238.130:59947)
0929 03:15:33.010376 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:33.010420 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:33.011945 (+ 1525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:33.012581 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:33.012591 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:33.013017 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:33.013658 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:33.013678 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:33.014506 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:33.014510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:33.015696 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:33.015707 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:33.016179 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:33.016183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:33.016352 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:33.016928 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:33.016946 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:33.018479 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:33.020170 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:33.020177 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:33.020195 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:33.020465 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:33.020726 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:33.020729 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:33.020732 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:33.020889 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:33.021271 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:33.021280 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:33.021549 (+ 269us) client_negotiation.cc:769] Sending connection context
0929 03:15:33.021730 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 03:15:33.022046 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.131@KRBTEST.COM" created.
I20260929 03:15:33.024359 17657 heartbeater.cc:347] Connected to a master server at 127.13.238.190:45743
I20260929 03:15:33.024603 17657 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:33.025107 17657 heartbeater.cc:511] Master 127.13.238.190:45743 requested a full tablet report, sending...
I20260929 03:15:33.026177 17332 ts_manager.cc:194] Registered new tserver with Master: ea9af098f2354ba890608a55419c6424 (127.13.238.130:33707)
I20260929 03:15:33.026767 17332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.130@KRBTEST.COM'} at 127.13.238.130:59947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
I20260929 03:15:33.039201 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:0
--local_ip_for_outbound_sockets=127.13.238.131
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:45743
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:33.149675 17662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:33.149943 17662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:33.149991 17662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:33.154048 17662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:33.154116 17662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:33.154186 17662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:15:33.159479 17662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:45743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:33.160563 17662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:33.161602 17662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:33.171314 17667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:33.171284 17668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:33.171355 17662 server_base.cc:1034] running on GCE node
W20260929 03:15:33.171303 17670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:33.172004 17662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:33.172673 17662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:33.173843 17662 hybrid_clock.cc:648] HybridClock initialized: now 1790651733173808 us; error 43 us; skew 500 ppm
Sep 29 03:15:33 dist-test-slave-nbjg krb5kdc[17287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651733, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:33.176549 17662 init.cc:377] Logged in from keytab as kudu/127.13.238.131@KRBTEST.COM (short username kudu)
I20260929 03:15:33.177413 17662 webserver.cc:460] Webserver started at http://127.13.238.131:39501/ using document root <none> and password file <none>
I20260929 03:15:33.177968 17662 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:33.178032 17662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:33.178249 17662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:33.180454 17662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/data/instance:
uuid: "344d2716fe89474195c38710f9626dc4"
format_stamp: "Formatted at 2026-09-29 03:15:33 on dist-test-slave-nbjg"
server_key: "9554d065768db285433aca5d731c97a3"
server_key_iv: "325e4590b782fae4faad1caea0e6240c"
server_key_version: "encryptionkey@0"
I20260929 03:15:33.180999 17662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance:
uuid: "344d2716fe89474195c38710f9626dc4"
format_stamp: "Formatted at 2026-09-29 03:15:33 on dist-test-slave-nbjg"
server_key: "9554d065768db285433aca5d731c97a3"
server_key_iv: "325e4590b782fae4faad1caea0e6240c"
server_key_version: "encryptionkey@0"
I20260929 03:15:33.184671 17662 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 03:15:33.187952 17677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:33.189284 17662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:15:33.189431 17662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "344d2716fe89474195c38710f9626dc4"
format_stamp: "Formatted at 2026-09-29 03:15:33 on dist-test-slave-nbjg"
server_key: "9554d065768db285433aca5d731c97a3"
server_key_iv: "325e4590b782fae4faad1caea0e6240c"
server_key_version: "encryptionkey@0"
I20260929 03:15:33.189528 17662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:33.201715 17662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:33.205435 17662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:33.205628 17662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:33.206346 17662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:33.207286 17662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:33.207340 17662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:33.207396 17662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:33.207420 17662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:33.215590 17662 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:37407
I20260929 03:15:33.215610 17790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:37407 every 8 connection(s)
I20260929 03:15:33.216470 17662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:15:33.225221 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 17662
I20260929 03:15:33.225347 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance
Sep 29 03:15:33 dist-test-slave-nbjg krb5kdc[17287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651733, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:33.225653 14266 external_mini_cluster.cc:1428] Setting key bf7efa4f5ca798af6910e0775936bd89
I20260929 03:15:33.229566 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:33.218246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:46647 (local address 127.13.238.190:45743)
0929 03:15:33.218378 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:33.218383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:33.219253 (+ 870us) server_negotiation.cc:407] Connection header received
0929 03:15:33.220572 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:33.220575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:33.220624 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:33.220697 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:33.221792 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:33.222358 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:33.223427 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:33.223633 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:33.223704 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:33.226247 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:33.226252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:33.226254 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:33.226282 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:33.227797 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:33.228220 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:33.228222 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:33.228223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:33.228292 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:33.228613 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:33.228617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:33.228619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:33.228805 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:33.228882 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:33.229295 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 03:15:33.229387 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:15:33.230345 17793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:33.218674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:45743 (local address 127.13.238.131:46647)
0929 03:15:33.219110 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:33.219151 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:33.220371 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:33.220850 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:33.220857 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:33.221196 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:33.221681 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:33.221695 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:33.222460 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:33.222463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:33.223337 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:33.223346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:33.223738 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:33.223740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:33.223875 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:33.224406 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:33.224422 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:33.226012 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:33.227892 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:33.227896 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:33.227908 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:33.228144 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:33.228353 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:33.228355 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:33.228356 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:33.228483 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:33.228885 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:33.228890 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:33.229111 (+ 221us) client_negotiation.cc:769] Sending connection context
0929 03:15:33.229259 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 03:15:33.229501 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":100,"threads_started":1}
I20260929 03:15:33.231328 17791 heartbeater.cc:347] Connected to a master server at 127.13.238.190:45743
I20260929 03:15:33.231545 17791 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:33.231990 17791 heartbeater.cc:511] Master 127.13.238.190:45743 requested a full tablet report, sending...
I20260929 03:15:33.233017 17332 ts_manager.cc:194] Registered new tserver with Master: 344d2716fe89474195c38710f9626dc4 (127.13.238.131:37407)
I20260929 03:15:33.233574 17332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.131@KRBTEST.COM'} at 127.13.238.131:46647
I20260929 03:15:33.239409 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:15:33 dist-test-slave-nbjg krb5kdc[17287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651733, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 03:15:33 dist-test-slave-nbjg krb5kdc[17287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651733, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:33.264037 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:33.255853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56394 (local address 127.13.238.190:45743)
0929 03:15:33.256019 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:33.256024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:33.256111 (+ 87us) server_negotiation.cc:407] Connection header received
0929 03:15:33.256257 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:33.256261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:33.256310 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:33.256382 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:33.257089 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:33.257766 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:33.258632 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:33.258853 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:33.258941 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:33.261175 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:33.261179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:33.261181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:33.261217 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:33.262424 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:33.262842 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:33.262846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:33.262848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:33.262926 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:33.263162 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:33.263165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:33.263167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:33.263410 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:33.263529 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:33.263764 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 03:15:33.263879 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 03:15:33.264528 17332 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56394
I20260929 03:15:33.265378 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 17394
I20260929 03:15:33.270569 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 17528
I20260929 03:15:33.275564 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 17662
I20260929 03:15:33.279887 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 17300
2026-09-29T03:15:33Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2026 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:15:33 dist-test-slave-nbjg krb5kdc[17803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:15:33 dist-test-slave-nbjg krb5kdc[17803](info): set up 2 sockets
Sep 29 03:15:33 dist-test-slave-nbjg krb5kdc[17803](info): commencing operation
krb5kdc: starting...
W20260929 03:15:34.234393 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:15:34 dist-test-slave-nbjg krb5kdc[17803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:15:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:15:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:15:34.347012 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:35715
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:40641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:35715
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:34.454805 17816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:34.455075 17816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:34.455130 17816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:34.459134 17816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:34.459192 17816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:34.459218 17816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:34.459237 17816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:34.459259 17816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:15:34.464450 17816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:40641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:35715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:35715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:34.465477 17816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:34.466514 17816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:34.474644 17821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:34.474665 17822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:34.474713 17816 server_base.cc:1034] running on GCE node
W20260929 03:15:34.474665 17824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:34.475414 17816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:34.476483 17816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:34.477646 17816 hybrid_clock.cc:648] HybridClock initialized: now 1790651734477632 us; error 24 us; skew 500 ppm
Sep 29 03:15:34 dist-test-slave-nbjg krb5kdc[17803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651734, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:34.487177 17816 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:15:34.488117 17816 webserver.cc:460] Webserver started at http://127.13.238.190:39641/ using document root <none> and password file <none>
I20260929 03:15:34.488692 17816 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:34.488761 17816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:34.488968 17816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:34.491089 17816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "a86a40a4bdf345f0a7997827e9273658"
format_stamp: "Formatted at 2026-09-29 03:15:34 on dist-test-slave-nbjg"
server_key: "94c978464f97f5ea9eea548147805830"
server_key_iv: "2bf064657b81786c7ad5ecb19cdd7114"
server_key_version: "encryptionkey@0"
I20260929 03:15:34.491631 17816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "a86a40a4bdf345f0a7997827e9273658"
format_stamp: "Formatted at 2026-09-29 03:15:34 on dist-test-slave-nbjg"
server_key: "94c978464f97f5ea9eea548147805830"
server_key_iv: "2bf064657b81786c7ad5ecb19cdd7114"
server_key_version: "encryptionkey@0"
I20260929 03:15:34.495314 17816 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:15:34.497661 17831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:34.498929 17816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:34.499045 17816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "a86a40a4bdf345f0a7997827e9273658"
format_stamp: "Formatted at 2026-09-29 03:15:34 on dist-test-slave-nbjg"
server_key: "94c978464f97f5ea9eea548147805830"
server_key_iv: "2bf064657b81786c7ad5ecb19cdd7114"
server_key_version: "encryptionkey@0"
I20260929 03:15:34.499135 17816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:34.534559 17816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:34.538072 17816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:34.538298 17816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:34.545396 17816 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:35715
I20260929 03:15:34.545398 17883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:35715 every 8 connection(s)
I20260929 03:15:34.546398 17816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:15:34.549666 17884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:34.552860 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 17816
I20260929 03:15:34.552984 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:15:34.553268 14266 external_mini_cluster.cc:1428] Setting key bee3526c65bddfc0b4c07eab6daa721a
I20260929 03:15:34.556712 17884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a86a40a4bdf345f0a7997827e9273658: Bootstrap starting.
I20260929 03:15:34.559574 17884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a86a40a4bdf345f0a7997827e9273658: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:34.560560 17884 log.cc:824] T 00000000000000000000000000000000 P a86a40a4bdf345f0a7997827e9273658: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:34.563120 17884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a86a40a4bdf345f0a7997827e9273658: No bootstrap required, opened a new log
Sep 29 03:15:34 dist-test-slave-nbjg krb5kdc[17803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:34.567202 17884 raft_consensus.cc:348] T 00000000000000000000000000000000 P a86a40a4bdf345f0a7997827e9273658 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a86a40a4bdf345f0a7997827e9273658" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 35715 } }
I20260929 03:15:34.567550 17884 raft_consensus.cc:374] T 00000000000000000000000000000000 P a86a40a4bdf345f0a7997827e9273658 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:34.567639 17884 raft_consensus.cc:729] T 00000000000000000000000000000000 P a86a40a4bdf345f0a7997827e9273658 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a86a40a4bdf345f0a7997827e9273658, State: Initialized, Role: FOLLOWER
I20260929 03:15:34.568213 17884 consensus_queue.cc:260] T 00000000000000000000000000000000 P a86a40a4bdf345f0a7997827e9273658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a86a40a4bdf345f0a7997827e9273658" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 35715 } }
I20260929 03:15:34.568394 17884 raft_consensus.cc:388] T 00000000000000000000000000000000 P a86a40a4bdf345f0a7997827e9273658 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:15:34.568509 17884 raft_consensus.cc:482] T 00000000000000000000000000000000 P a86a40a4bdf345f0a7997827e9273658 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:15:34.568650 17884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a86a40a4bdf345f0a7997827e9273658 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:34.569872 17884 raft_consensus.cc:504] T 00000000000000000000000000000000 P a86a40a4bdf345f0a7997827e9273658 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a86a40a4bdf345f0a7997827e9273658" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 35715 } }
I20260929 03:15:34.570323 17884 leader_election.cc:302] T 00000000000000000000000000000000 P a86a40a4bdf345f0a7997827e9273658 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a86a40a4bdf345f0a7997827e9273658; no voters:
I20260929 03:15:34.570753 17884 leader_election.cc:288] T 00000000000000000000000000000000 P a86a40a4bdf345f0a7997827e9273658 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:15:34.570874 17889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a86a40a4bdf345f0a7997827e9273658 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:34.571156 17889 raft_consensus.cc:686] T 00000000000000000000000000000000 P a86a40a4bdf345f0a7997827e9273658 [term 1 LEADER]: Becoming Leader. State: Replica: a86a40a4bdf345f0a7997827e9273658, State: Running, Role: LEADER
I20260929 03:15:34.571557 17889 consensus_queue.cc:237] T 00000000000000000000000000000000 P a86a40a4bdf345f0a7997827e9273658 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a86a40a4bdf345f0a7997827e9273658" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 35715 } }
I20260929 03:15:34.572085 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:34.554868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39726 (local address 127.13.238.190:35715)
0929 03:15:34.555361 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:34.555381 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:34.555662 (+ 281us) server_negotiation.cc:407] Connection header received
0929 03:15:34.556614 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:34.556639 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:34.557103 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:34.557482 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:34.558405 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:34.559970 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:34.561036 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:34.561649 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:34.561802 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:34.564985 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:34.564991 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:34.565007 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:34.565037 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:34.567972 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:34.568567 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:34.568574 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:34.568581 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:34.568741 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:34.569078 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:34.569081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:34.569083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:34.569663 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:34.569887 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:34.570638 (+ 751us) server_negotiation.cc:299] Negotiation successful
0929 03:15:34.570994 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":135,"threads_started":1}
I20260929 03:15:34.572392 17884 sys_catalog.cc:564] T 00000000000000000000000000000000 P a86a40a4bdf345f0a7997827e9273658 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:34.575659 17891 sys_catalog.cc:455] T 00000000000000000000000000000000 P a86a40a4bdf345f0a7997827e9273658 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a86a40a4bdf345f0a7997827e9273658" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a86a40a4bdf345f0a7997827e9273658" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 35715 } } }
I20260929 03:15:34.576443 17890 sys_catalog.cc:455] T 00000000000000000000000000000000 P a86a40a4bdf345f0a7997827e9273658 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a86a40a4bdf345f0a7997827e9273658. Latest consensus state: current_term: 1 leader_uuid: "a86a40a4bdf345f0a7997827e9273658" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a86a40a4bdf345f0a7997827e9273658" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 35715 } } }
I20260929 03:15:34.576900 17891 sys_catalog.cc:458] T 00000000000000000000000000000000 P a86a40a4bdf345f0a7997827e9273658 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:34.577003 17890 sys_catalog.cc:458] T 00000000000000000000000000000000 P a86a40a4bdf345f0a7997827e9273658 [sys.catalog]: This master's current role is: LEADER
W20260929 03:15:34.578679 17903 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a86a40a4bdf345f0a7997827e9273658: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:15:34.578792 17903 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:15:34.580193 17906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:15:34.580940 17906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:15:34.586787 17906 catalog_manager.cc:1347] Generated new cluster ID: 9b09d589499b41c3a39eb8175133750a
I20260929 03:15:34.586848 17906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:15:34.597474 17906 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:15:34.598711 17906 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:15:34.606106 17906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a86a40a4bdf345f0a7997827e9273658: Generated new TSK 0
I20260929 03:15:34.606887 17906 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
I20260929 03:15:34.639225 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:0
--local_ip_for_outbound_sockets=127.13.238.129
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:35715
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:40641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:34.750448 17910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:34.750730 17910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:34.750772 17910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:34.754845 17910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:34.754902 17910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:34.754961 17910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:15:34.760198 17910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:40641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:35715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:34.761219 17910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:34.762254 17910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:34.771785 17918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:34.771785 17915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:34.771970 17910 server_base.cc:1034] running on GCE node
W20260929 03:15:34.771903 17916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:34.772511 17910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:34.773236 17910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:34.774413 17910 hybrid_clock.cc:648] HybridClock initialized: now 1790651734774387 us; error 42 us; skew 500 ppm
Sep 29 03:15:34 dist-test-slave-nbjg krb5kdc[17803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651734, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:34.781481 17910 init.cc:377] Logged in from keytab as kudu/127.13.238.129@KRBTEST.COM (short username kudu)
I20260929 03:15:34.782460 17910 webserver.cc:460] Webserver started at http://127.13.238.129:36099/ using document root <none> and password file <none>
I20260929 03:15:34.783030 17910 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:34.783097 17910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:34.783288 17910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:34.785416 17910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-0/data/instance:
uuid: "bdeb30042f8b4646afc78413ab96b304"
format_stamp: "Formatted at 2026-09-29 03:15:34 on dist-test-slave-nbjg"
server_key: "c8c58bcc07adb73e49f15d8f4d97415d"
server_key_iv: "83e36d697924163e78c369cdac82f6ae"
server_key_version: "encryptionkey@0"
I20260929 03:15:34.785966 17910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance:
uuid: "bdeb30042f8b4646afc78413ab96b304"
format_stamp: "Formatted at 2026-09-29 03:15:34 on dist-test-slave-nbjg"
server_key: "c8c58bcc07adb73e49f15d8f4d97415d"
server_key_iv: "83e36d697924163e78c369cdac82f6ae"
server_key_version: "encryptionkey@0"
I20260929 03:15:34.789667 17910 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 03:15:34.792152 17925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:34.793380 17910 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:15:34.793501 17910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "bdeb30042f8b4646afc78413ab96b304"
format_stamp: "Formatted at 2026-09-29 03:15:34 on dist-test-slave-nbjg"
server_key: "c8c58bcc07adb73e49f15d8f4d97415d"
server_key_iv: "83e36d697924163e78c369cdac82f6ae"
server_key_version: "encryptionkey@0"
I20260929 03:15:34.793594 17910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:34.829068 17910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:34.832010 17910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:34.832190 17910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:34.832818 17910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:34.833678 17910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:34.833725 17910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:34.833789 17910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:34.833813 17910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:34.842001 17910 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:33791
I20260929 03:15:34.842034 18038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:33791 every 8 connection(s)
I20260929 03:15:34.842981 17910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:15:34.845448 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 17910
I20260929 03:15:34.845561 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance
I20260929 03:15:34.845829 14266 external_mini_cluster.cc:1428] Setting key e2efa1e62d879d1463db77a567bd6b77
Sep 29 03:15:34 dist-test-slave-nbjg krb5kdc[17803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651734, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.238.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.130@KRBTEST.COM" created.
I20260929 03:15:34.857167 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:34.844847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:35339 (local address 127.13.238.190:35715)
0929 03:15:34.844985 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:34.844988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:34.845887 (+ 899us) server_negotiation.cc:407] Connection header received
0929 03:15:34.847403 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:34.847406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:34.847457 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:34.847551 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:34.848778 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:34.849311 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:34.850945 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:34.851214 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:34.851296 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:34.853886 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:34.853889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:34.853891 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:34.853915 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:34.855285 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:34.855753 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:34.855756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:34.855757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:34.855843 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:34.856152 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:34.856154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:34.856155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:34.856322 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:34.856396 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:34.856915 (+ 519us) server_negotiation.cc:299] Negotiation successful
0929 03:15:34.857021 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:15:34.858044 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:34.845235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:35715 (local address 127.13.238.129:35339)
0929 03:15:34.845701 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:34.845765 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:34.847194 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:34.847691 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:34.847697 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:34.848071 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:34.848614 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:34.848635 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:34.849435 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:34.849438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:34.850811 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:34.850821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:34.851363 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:34.851367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:34.851574 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:34.852147 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:34.852168 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:34.853728 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:34.855391 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:34.855397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:34.855412 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:34.855658 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:34.855946 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:34.855949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:34.855950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:34.856081 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:34.856395 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:34.856400 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:34.856683 (+ 283us) client_negotiation.cc:769] Sending connection context
0929 03:15:34.856861 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 03:15:34.857178 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":95,"threads_started":1}
I20260929 03:15:34.859196 18039 heartbeater.cc:347] Connected to a master server at 127.13.238.190:35715
I20260929 03:15:34.859467 18039 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:34.859951 18039 heartbeater.cc:511] Master 127.13.238.190:35715 requested a full tablet report, sending...
I20260929 03:15:34.861613 17848 ts_manager.cc:194] Registered new tserver with Master: bdeb30042f8b4646afc78413ab96b304 (127.13.238.129:33791)
I20260929 03:15:34.863226 17848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.129@KRBTEST.COM'} at 127.13.238.129:35339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
I20260929 03:15:34.870945 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:0
--local_ip_for_outbound_sockets=127.13.238.130
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:35715
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:40641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:34.979065 18044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:34.979343 18044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:34.979393 18044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:34.983449 18044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:34.983507 18044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:34.983582 18044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:15:34.988934 18044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:40641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:35715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:34.989991 18044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:34.991039 18044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:15:35.000815 18044 server_base.cc:1034] running on GCE node
W20260929 03:15:35.000835 18049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:35.000799 18050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:35.000787 18052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:35.001511 18044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:35.002147 18044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:35.003331 18044 hybrid_clock.cc:648] HybridClock initialized: now 1790651735003304 us; error 39 us; skew 500 ppm
Sep 29 03:15:35 dist-test-slave-nbjg krb5kdc[17803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651735, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:35.006022 18044 init.cc:377] Logged in from keytab as kudu/127.13.238.130@KRBTEST.COM (short username kudu)
I20260929 03:15:35.006929 18044 webserver.cc:460] Webserver started at http://127.13.238.130:41161/ using document root <none> and password file <none>
I20260929 03:15:35.007453 18044 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:35.007512 18044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:35.007705 18044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:35.009688 18044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-1/data/instance:
uuid: "2a559061ad114421985557ea984f79cc"
format_stamp: "Formatted at 2026-09-29 03:15:35 on dist-test-slave-nbjg"
server_key: "ef4f59a893bf7b161d4173a52cecd370"
server_key_iv: "d575e548dd3a803c6b5b27cdef160c57"
server_key_version: "encryptionkey@0"
I20260929 03:15:35.010219 18044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance:
uuid: "2a559061ad114421985557ea984f79cc"
format_stamp: "Formatted at 2026-09-29 03:15:35 on dist-test-slave-nbjg"
server_key: "ef4f59a893bf7b161d4173a52cecd370"
server_key_iv: "d575e548dd3a803c6b5b27cdef160c57"
server_key_version: "encryptionkey@0"
I20260929 03:15:35.013768 18044 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:15:35.016166 18059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:35.017381 18044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:15:35.017501 18044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "2a559061ad114421985557ea984f79cc"
format_stamp: "Formatted at 2026-09-29 03:15:35 on dist-test-slave-nbjg"
server_key: "ef4f59a893bf7b161d4173a52cecd370"
server_key_iv: "d575e548dd3a803c6b5b27cdef160c57"
server_key_version: "encryptionkey@0"
I20260929 03:15:35.017596 18044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:35.043159 18044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:35.046288 18044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:35.046463 18044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:35.047045 18044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:35.047853 18044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:35.047909 18044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:35.047961 18044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:35.047984 18044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:35.055943 18044 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:45853
I20260929 03:15:35.055984 18172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:45853 every 8 connection(s)
I20260929 03:15:35.056788 18044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:15:35 dist-test-slave-nbjg krb5kdc[17803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651735, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:35.066672 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 18044
I20260929 03:15:35.066772 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance
I20260929 03:15:35.067028 14266 external_mini_cluster.cc:1428] Setting key c5657382b995513c376b598f06c6f95a
I20260929 03:15:35.069504 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:35.058574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:37683 (local address 127.13.238.190:35715)
0929 03:15:35.058692 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:35.058696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:35.059488 (+ 792us) server_negotiation.cc:407] Connection header received
0929 03:15:35.060777 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:35.060779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:35.060828 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:35.060902 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:35.061999 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:35.062591 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:35.063656 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:35.063873 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:35.063982 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:35.066129 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:35.066132 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:35.066134 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:35.066159 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:35.067608 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:35.068078 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:35.068082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:35.068084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:35.068164 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:35.068480 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:35.068483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:35.068485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:35.068664 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:35.068803 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:35.069231 (+ 428us) server_negotiation.cc:299] Negotiation successful
0929 03:15:35.069346 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:15:35.070317 18175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:35.058908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:35715 (local address 127.13.238.130:37683)
0929 03:15:35.059325 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:35.059367 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:35.060590 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:35.061058 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:35.061064 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:35.061394 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:35.061870 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:35.061883 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:35.062683 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:35.062686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:35.063560 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:35.063568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:35.064018 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:35.064020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:35.064152 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:35.064641 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:35.064657 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:35.065958 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:35.067716 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:35.067721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:35.067737 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:35.067990 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:35.068258 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:35.068261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:35.068263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:35.068395 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:35.068760 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:35.068766 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:35.069011 (+ 245us) client_negotiation.cc:769] Sending connection context
0929 03:15:35.069173 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 03:15:35.069487 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":104,"threads_started":1}
I20260929 03:15:35.071378 18173 heartbeater.cc:347] Connected to a master server at 127.13.238.190:35715
I20260929 03:15:35.071631 18173 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:35.072170 18173 heartbeater.cc:511] Master 127.13.238.190:35715 requested a full tablet report, sending...
I20260929 03:15:35.073324 17848 ts_manager.cc:194] Registered new tserver with Master: 2a559061ad114421985557ea984f79cc (127.13.238.130:45853)
I20260929 03:15:35.073908 17848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.130@KRBTEST.COM'} at 127.13.238.130:37683
WARNING: no policy specified for kudu/127.13.238.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
I20260929 03:15:35.092885 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:0
--local_ip_for_outbound_sockets=127.13.238.131
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:35715
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:40641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:35.206499 18178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:35.206882 18178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:35.206969 18178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:35.213227 18178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:35.213302 18178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:35.213394 18178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:15:35.220369 18178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:40641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:35715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:35.221457 18178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:35.222514 18178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:35.231657 18183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:35.231755 18178 server_base.cc:1034] running on GCE node
W20260929 03:15:35.231920 18184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:35.231739 18186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:35.232416 18178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:35.233068 18178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:35.234241 18178 hybrid_clock.cc:648] HybridClock initialized: now 1790651735234216 us; error 36 us; skew 500 ppm
Sep 29 03:15:35 dist-test-slave-nbjg krb5kdc[17803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651735, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:35.236893 18178 init.cc:377] Logged in from keytab as kudu/127.13.238.131@KRBTEST.COM (short username kudu)
I20260929 03:15:35.237717 18178 webserver.cc:460] Webserver started at http://127.13.238.131:40689/ using document root <none> and password file <none>
I20260929 03:15:35.238337 18178 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:35.238405 18178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:35.238579 18178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:35.240618 18178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-2/data/instance:
uuid: "b63c450c577a431dbd10248cfc4c6313"
format_stamp: "Formatted at 2026-09-29 03:15:35 on dist-test-slave-nbjg"
server_key: "3b03f6694e62780d202b24c781cc3c69"
server_key_iv: "41d016a87c6d7aaf0969d833d8c1b59e"
server_key_version: "encryptionkey@0"
I20260929 03:15:35.241142 18178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance:
uuid: "b63c450c577a431dbd10248cfc4c6313"
format_stamp: "Formatted at 2026-09-29 03:15:35 on dist-test-slave-nbjg"
server_key: "3b03f6694e62780d202b24c781cc3c69"
server_key_iv: "41d016a87c6d7aaf0969d833d8c1b59e"
server_key_version: "encryptionkey@0"
I20260929 03:15:35.244858 18178 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:15:35.247275 18193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:35.248445 18178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:35.248540 18178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "b63c450c577a431dbd10248cfc4c6313"
format_stamp: "Formatted at 2026-09-29 03:15:35 on dist-test-slave-nbjg"
server_key: "3b03f6694e62780d202b24c781cc3c69"
server_key_iv: "41d016a87c6d7aaf0969d833d8c1b59e"
server_key_version: "encryptionkey@0"
I20260929 03:15:35.248636 18178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:35.282177 18178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:35.285427 18178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:35.285604 18178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:35.286249 18178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:35.287099 18178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:35.287141 18178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:35.287184 18178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:35.287212 18178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:35.295421 18178 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:33715
I20260929 03:15:35.295495 18306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:33715 every 8 connection(s)
I20260929 03:15:35.296373 18178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:15:35.299351 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 18178
I20260929 03:15:35.299479 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance
I20260929 03:15:35.299790 14266 external_mini_cluster.cc:1428] Setting key 1129dc43644852270a010eedabe61643
Sep 29 03:15:35 dist-test-slave-nbjg krb5kdc[17803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651735, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:35.310551 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:35.298337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:53505 (local address 127.13.238.190:35715)
0929 03:15:35.298496 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:35.298501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:35.299592 (+ 1091us) server_negotiation.cc:407] Connection header received
0929 03:15:35.301224 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:35.301227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:35.301284 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:35.301361 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:35.302738 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:35.303317 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:35.304485 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:35.304711 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:35.304801 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:35.307196 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:35.307199 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:35.307201 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:35.307227 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:35.308516 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:35.309017 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:35.309019 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:35.309021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:35.309102 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:35.309429 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:35.309433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:35.309435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:35.309620 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:35.309707 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:35.310249 (+ 542us) server_negotiation.cc:299] Negotiation successful
0929 03:15:35.310381 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:15:35.311363 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:35.298851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:35715 (local address 127.13.238.131:53505)
0929 03:15:35.299397 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:35.299484 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:35.300991 (+ 1507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:35.301573 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:35.301581 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:35.301979 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:35.302603 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:35.302620 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:35.303443 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:35.303446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:35.304363 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:35.304374 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:35.304855 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:35.304859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:35.305030 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:35.305571 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:35.305588 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:35.307023 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:35.308627 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:35.308634 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:35.308651 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:35.308906 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:35.309183 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:35.309186 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:35.309189 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:35.309331 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:35.309721 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:35.309728 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:35.309981 (+ 253us) client_negotiation.cc:769] Sending connection context
0929 03:15:35.310147 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 03:15:35.310527 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":106,"threads_started":1}
I20260929 03:15:35.312525 18307 heartbeater.cc:347] Connected to a master server at 127.13.238.190:35715
I20260929 03:15:35.312793 18307 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:35.313407 18307 heartbeater.cc:511] Master 127.13.238.190:35715 requested a full tablet report, sending...
I20260929 03:15:35.314513 17848 ts_manager.cc:194] Registered new tserver with Master: b63c450c577a431dbd10248cfc4c6313 (127.13.238.131:33715)
I20260929 03:15:35.315075 17848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.131@KRBTEST.COM'} at 127.13.238.131:53505
I20260929 03:15:35.325618 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:15:35 dist-test-slave-nbjg krb5kdc[17803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:15:35 dist-test-slave-nbjg krb5kdc[17803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.129@KRBTEST.COM
I20260929 03:15:35.350490 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:35.341176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54790 (local address 127.13.238.129:33791)
0929 03:15:35.341489 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:35.341504 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:35.341529 (+ 25us) server_negotiation.cc:407] Connection header received
0929 03:15:35.341581 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:35.341587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:35.341787 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:35.341907 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:35.342608 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:35.343486 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:35.344295 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:35.344672 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:35.344766 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:35.347133 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:35.347143 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:35.347159 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:35.347184 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:35.348721 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:35.349129 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:35.349135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:35.349136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:35.349214 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:35.349464 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:35.349467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:35.349469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:35.349710 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:35.349932 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:35.350110 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 03:15:35.350298 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":108,"threads_started":1}
Sep 29 03:15:35 dist-test-slave-nbjg krb5kdc[17803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:35.360985 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:35.353476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39738 (local address 127.13.238.190:35715)
0929 03:15:35.353626 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:35.353629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:35.353644 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:15:35.353753 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:35.353755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:35.353796 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:35.353862 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:35.354502 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:35.355047 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:35.355749 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:35.355959 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:35.356034 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:35.358253 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:35.358258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:35.358261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:35.358291 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:35.359600 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:35.359967 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:35.359970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:35.359971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:35.360040 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:35.360219 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:35.360221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:35.360222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:35.360434 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:35.360533 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:35.360686 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 03:15:35.360821 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 03:15:35.361399 17848 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39738
I20260929 03:15:35.362002 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 17910
I20260929 03:15:35.368294 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 18044
I20260929 03:15:35.373476 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 18178
I20260929 03:15:35.378334 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 17816
2026-09-29T03:15:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2098 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:15:35 dist-test-slave-nbjg krb5kdc[18319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:15:35 dist-test-slave-nbjg krb5kdc[18319](info): set up 2 sockets
Sep 29 03:15:35 dist-test-slave-nbjg krb5kdc[18319](info): commencing operation
krb5kdc: starting...
W20260929 03:15:36.334381 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:15:36 dist-test-slave-nbjg krb5kdc[18319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:15:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:15:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:15:36.448457 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:38687
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:38687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:36.559055 18332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:36.559325 18332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:36.559383 18332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:36.563371 18332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:36.563428 18332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:36.563453 18332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:36.563473 18332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:36.563494 18332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:15:36.568665 18332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:38687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:38687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:36.569690 18332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:36.570748 18332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:36.578691 18338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:36.578716 18337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:36.578701 18340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:36.578742 18332 server_base.cc:1034] running on GCE node
I20260929 03:15:36.579505 18332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:36.580646 18332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:36.581832 18332 hybrid_clock.cc:648] HybridClock initialized: now 1790651736581822 us; error 46 us; skew 500 ppm
Sep 29 03:15:36 dist-test-slave-nbjg krb5kdc[18319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651736, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:36.584543 18332 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:15:36.585292 18332 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:36.585361 18332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:36.585561 18332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:36.587683 18332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "64a3726e22be48ab8258c9d22f668124"
format_stamp: "Formatted at 2026-09-29 03:15:36 on dist-test-slave-nbjg"
server_key: "8a39c6846b92e44217d883892a243baa"
server_key_iv: "10834f09d700d4791ef3e46bb916d5b2"
server_key_version: "encryptionkey@0"
I20260929 03:15:36.588228 18332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "64a3726e22be48ab8258c9d22f668124"
format_stamp: "Formatted at 2026-09-29 03:15:36 on dist-test-slave-nbjg"
server_key: "8a39c6846b92e44217d883892a243baa"
server_key_iv: "10834f09d700d4791ef3e46bb916d5b2"
server_key_version: "encryptionkey@0"
I20260929 03:15:36.591987 18332 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 03:15:36.594436 18346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:36.595768 18332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:15:36.595888 18332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "64a3726e22be48ab8258c9d22f668124"
format_stamp: "Formatted at 2026-09-29 03:15:36 on dist-test-slave-nbjg"
server_key: "8a39c6846b92e44217d883892a243baa"
server_key_iv: "10834f09d700d4791ef3e46bb916d5b2"
server_key_version: "encryptionkey@0"
I20260929 03:15:36.595996 18332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:36.624662 18332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:36.627756 18332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:36.627945 18332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:36.634862 18332 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:38687
I20260929 03:15:36.635087 18398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:38687 every 8 connection(s)
I20260929 03:15:36.635536 18332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:15:36.638278 18399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:36.643710 18399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a3726e22be48ab8258c9d22f668124: Bootstrap starting.
I20260929 03:15:36.644765 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 18332
I20260929 03:15:36.644887 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:15:36.645161 14266 external_mini_cluster.cc:1428] Setting key a013ecae41b8ce683df2a9a3000e1180
I20260929 03:15:36.646636 18399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64a3726e22be48ab8258c9d22f668124: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:36.647367 18399 log.cc:824] T 00000000000000000000000000000000 P 64a3726e22be48ab8258c9d22f668124: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:36.649564 18399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a3726e22be48ab8258c9d22f668124: No bootstrap required, opened a new log
Sep 29 03:15:36 dist-test-slave-nbjg krb5kdc[18319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:36.653810 18399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64a3726e22be48ab8258c9d22f668124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a3726e22be48ab8258c9d22f668124" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 38687 } }
I20260929 03:15:36.654066 18399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64a3726e22be48ab8258c9d22f668124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:36.654122 18399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64a3726e22be48ab8258c9d22f668124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64a3726e22be48ab8258c9d22f668124, State: Initialized, Role: FOLLOWER
I20260929 03:15:36.654614 18399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64a3726e22be48ab8258c9d22f668124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a3726e22be48ab8258c9d22f668124" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 38687 } }
I20260929 03:15:36.654790 18399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64a3726e22be48ab8258c9d22f668124 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:15:36.654860 18399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64a3726e22be48ab8258c9d22f668124 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:15:36.654959 18399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64a3726e22be48ab8258c9d22f668124 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:36.656054 18399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64a3726e22be48ab8258c9d22f668124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a3726e22be48ab8258c9d22f668124" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 38687 } }
I20260929 03:15:36.656424 18399 leader_election.cc:302] T 00000000000000000000000000000000 P 64a3726e22be48ab8258c9d22f668124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64a3726e22be48ab8258c9d22f668124; no voters:
I20260929 03:15:36.656752 18399 leader_election.cc:288] T 00000000000000000000000000000000 P 64a3726e22be48ab8258c9d22f668124 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:15:36.656839 18404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64a3726e22be48ab8258c9d22f668124 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:36.657006 18404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64a3726e22be48ab8258c9d22f668124 [term 1 LEADER]: Becoming Leader. State: Replica: 64a3726e22be48ab8258c9d22f668124, State: Running, Role: LEADER
I20260929 03:15:36.657367 18404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64a3726e22be48ab8258c9d22f668124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a3726e22be48ab8258c9d22f668124" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 38687 } }
I20260929 03:15:36.658067 18399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64a3726e22be48ab8258c9d22f668124 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:36.658931 18405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a3726e22be48ab8258c9d22f668124 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64a3726e22be48ab8258c9d22f668124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a3726e22be48ab8258c9d22f668124" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 38687 } } }
I20260929 03:15:36.659395 18405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a3726e22be48ab8258c9d22f668124 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:36.659227 18406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a3726e22be48ab8258c9d22f668124 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64a3726e22be48ab8258c9d22f668124. Latest consensus state: current_term: 1 leader_uuid: "64a3726e22be48ab8258c9d22f668124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a3726e22be48ab8258c9d22f668124" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 38687 } } }
I20260929 03:15:36.659695 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:36.646343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35116 (local address 127.13.238.190:38687)
0929 03:15:36.646736 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:36.646756 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:36.646801 (+ 45us) server_negotiation.cc:407] Connection header received
0929 03:15:36.647432 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:36.647448 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:36.647866 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:36.648241 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:36.649047 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:36.650054 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:36.651208 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:36.651678 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:36.651792 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:36.654451 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:36.654457 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:36.654474 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:36.654506 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:36.656381 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:36.656810 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:36.656816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:36.656823 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:36.657007 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:36.657316 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:36.657319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:36.657321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:36.657778 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:36.657972 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:36.658320 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 03:15:36.658656 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":99,"threads_started":1}
I20260929 03:15:36.659736 18406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a3726e22be48ab8258c9d22f668124 [sys.catalog]: This master's current role is: LEADER
W20260929 03:15:36.663132 18420 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 64a3726e22be48ab8258c9d22f668124: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:15:36.663209 18420 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:15:36.663318 18413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:15:36.664000 18413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:15:36.669417 18413 catalog_manager.cc:1347] Generated new cluster ID: 6f0370505a684022b668ce5e64f112fb
I20260929 03:15:36.669469 18413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:15:36.677348 18413 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:15:36.678320 18413 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:15:36.684398 18413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64a3726e22be48ab8258c9d22f668124: Generated new TSK 0
I20260929 03:15:36.685216 18413 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
I20260929 03:15:36.717389 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:0
--local_ip_for_outbound_sockets=127.13.238.129
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:38687
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:36.846781 18425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:36.847064 18425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:36.847128 18425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:36.851284 18425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:36.851358 18425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:36.851444 18425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:15:36.857055 18425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:38687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:36.858250 18425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:36.859432 18425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:36.868697 18430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:36.868736 18425 server_base.cc:1034] running on GCE node
W20260929 03:15:36.868705 18433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:36.868690 18431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:36.869328 18425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:36.869939 18425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:36.871095 18425 hybrid_clock.cc:648] HybridClock initialized: now 1790651736871072 us; error 35 us; skew 500 ppm
Sep 29 03:15:36 dist-test-slave-nbjg krb5kdc[18319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651736, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:36.873709 18425 init.cc:377] Logged in from keytab as kudu/127.13.238.129@KRBTEST.COM (short username kudu)
I20260929 03:15:36.874414 18425 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:36.874476 18425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:36.874648 18425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:36.876586 18425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-0/data/instance:
uuid: "5f7403498aec4818b22305d21cc72c7f"
format_stamp: "Formatted at 2026-09-29 03:15:36 on dist-test-slave-nbjg"
server_key: "bba6e099f09fa34a10cd843ec696c186"
server_key_iv: "a0b1d90ef312d7b476ea001faeac6881"
server_key_version: "encryptionkey@0"
I20260929 03:15:36.877084 18425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance:
uuid: "5f7403498aec4818b22305d21cc72c7f"
format_stamp: "Formatted at 2026-09-29 03:15:36 on dist-test-slave-nbjg"
server_key: "bba6e099f09fa34a10cd843ec696c186"
server_key_iv: "a0b1d90ef312d7b476ea001faeac6881"
server_key_version: "encryptionkey@0"
I20260929 03:15:36.880681 18425 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 03:15:36.883033 18439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:36.884225 18425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:36.884317 18425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "5f7403498aec4818b22305d21cc72c7f"
format_stamp: "Formatted at 2026-09-29 03:15:36 on dist-test-slave-nbjg"
server_key: "bba6e099f09fa34a10cd843ec696c186"
server_key_iv: "a0b1d90ef312d7b476ea001faeac6881"
server_key_version: "encryptionkey@0"
I20260929 03:15:36.884397 18425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:36.897133 18425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:36.900261 18425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:36.900434 18425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:36.900969 18425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:36.901727 18425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:36.901779 18425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:36.901819 18425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:36.901839 18425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:36.909977 18425 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:36863
I20260929 03:15:36.909998 18552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:36863 every 8 connection(s)
I20260929 03:15:36.910605 18425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:15:36.914523 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 18425
I20260929 03:15:36.914654 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance
I20260929 03:15:36.914907 14266 external_mini_cluster.cc:1428] Setting key 918ccab3dab589603ae7ae14ecbcebac
Sep 29 03:15:36 dist-test-slave-nbjg krb5kdc[18319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651736, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.238.130@KRBTEST.COM; defaulting to no policy
I20260929 03:15:36.924410 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:36.912428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:46791 (local address 127.13.238.190:38687)
0929 03:15:36.912573 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:36.912577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:36.913453 (+ 876us) server_negotiation.cc:407] Connection header received
0929 03:15:36.914834 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:36.914837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:36.914900 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:36.914973 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:36.916319 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:36.916860 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:36.918016 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:36.918310 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:36.918400 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:36.920779 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:36.920783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:36.920787 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:36.920812 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:36.922393 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:36.922894 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:36.922898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:36.922900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:36.922982 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:36.923297 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:36.923301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:36.923303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:36.923491 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:36.923574 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:36.924154 (+ 580us) server_negotiation.cc:299] Negotiation successful
0929 03:15:36.924280 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:15:36.925319 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:36.912815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:38687 (local address 127.13.238.129:46791)
0929 03:15:36.913297 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:36.913342 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:36.914586 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:36.915157 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:36.915164 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:36.915604 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:36.916182 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:36.916201 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:36.916982 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:36.916985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:36.917891 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:36.917901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:36.918454 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:36.918458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:36.918623 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:36.919178 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:36.919196 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:36.920598 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:36.922509 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:36.922516 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:36.922534 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:36.922798 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:36.923063 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:36.923066 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:36.923068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:36.923218 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:36.923585 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:36.923593 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:36.923908 (+ 315us) client_negotiation.cc:769] Sending connection context
0929 03:15:36.924101 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 03:15:36.924406 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":121,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.130@KRBTEST.COM" created.
I20260929 03:15:36.926741 18553 heartbeater.cc:347] Connected to a master server at 127.13.238.190:38687
I20260929 03:15:36.927030 18553 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:36.927564 18553 heartbeater.cc:511] Master 127.13.238.190:38687 requested a full tablet report, sending...
I20260929 03:15:36.929414 18363 ts_manager.cc:194] Registered new tserver with Master: 5f7403498aec4818b22305d21cc72c7f (127.13.238.129:36863)
I20260929 03:15:36.931137 18363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.129@KRBTEST.COM'} at 127.13.238.129:46791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
I20260929 03:15:36.940666 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:0
--local_ip_for_outbound_sockets=127.13.238.130
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:38687
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:37.048839 18558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:37.049117 18558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:37.049163 18558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:37.053200 18558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:37.053275 18558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:37.053339 18558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:15:37.058902 18558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:38687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:37.060096 18558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:37.061148 18558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:37.070695 18563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:37.070747 18558 server_base.cc:1034] running on GCE node
W20260929 03:15:37.070729 18564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:37.070886 18566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:37.071272 18558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:37.071913 18558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:37.073065 18558 hybrid_clock.cc:648] HybridClock initialized: now 1790651737073045 us; error 35 us; skew 500 ppm
Sep 29 03:15:37 dist-test-slave-nbjg krb5kdc[18319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651737, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:37.075891 18558 init.cc:377] Logged in from keytab as kudu/127.13.238.130@KRBTEST.COM (short username kudu)
I20260929 03:15:37.076604 18558 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:37.076671 18558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:37.076874 18558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:37.078987 18558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-1/data/instance:
uuid: "6ce1686e36e046ecb7bc5599a726b928"
format_stamp: "Formatted at 2026-09-29 03:15:37 on dist-test-slave-nbjg"
server_key: "d325ac59675616eae283c7c0c853194c"
server_key_iv: "130a7adf1e74adfb2e8079f303e26f70"
server_key_version: "encryptionkey@0"
I20260929 03:15:37.079517 18558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance:
uuid: "6ce1686e36e046ecb7bc5599a726b928"
format_stamp: "Formatted at 2026-09-29 03:15:37 on dist-test-slave-nbjg"
server_key: "d325ac59675616eae283c7c0c853194c"
server_key_iv: "130a7adf1e74adfb2e8079f303e26f70"
server_key_version: "encryptionkey@0"
I20260929 03:15:37.083241 18558 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:15:37.085654 18572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:37.087016 18558 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:37.087124 18558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "6ce1686e36e046ecb7bc5599a726b928"
format_stamp: "Formatted at 2026-09-29 03:15:37 on dist-test-slave-nbjg"
server_key: "d325ac59675616eae283c7c0c853194c"
server_key_iv: "130a7adf1e74adfb2e8079f303e26f70"
server_key_version: "encryptionkey@0"
I20260929 03:15:37.087220 18558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:37.095952 18558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:37.098858 18558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:37.099012 18558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:37.099548 18558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:37.100323 18558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:37.100368 18558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:37.100418 18558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:37.100443 18558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:37.108551 18558 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:40529
I20260929 03:15:37.108582 18685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:40529 every 8 connection(s)
I20260929 03:15:37.109094 18558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:15:37.116468 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 18558
I20260929 03:15:37.116571 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance
I20260929 03:15:37.116832 14266 external_mini_cluster.cc:1428] Setting key f90f86734d7c3cc0c8a9edeae2793366
Sep 29 03:15:37 dist-test-slave-nbjg krb5kdc[18319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651737, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:37.122248 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:37.110765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:51811 (local address 127.13.238.190:38687)
0929 03:15:37.110929 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:37.110933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:37.111817 (+ 884us) server_negotiation.cc:407] Connection header received
0929 03:15:37.113060 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:37.113062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:37.113108 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:37.113178 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:37.114343 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.114858 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.115972 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.116175 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.116261 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:37.118617 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:37.118621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:37.118624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:37.118653 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:37.120056 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:37.120613 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:37.120615 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:37.120617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:37.120683 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:37.121058 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:37.121059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:37.121060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:37.121220 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:37.121291 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:37.121884 (+ 593us) server_negotiation.cc:299] Negotiation successful
0929 03:15:37.122016 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 03:15:37.123046 18688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:37.111142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:38687 (local address 127.13.238.130:51811)
0929 03:15:37.111615 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:37.111657 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:37.112873 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:37.113320 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:37.113327 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:37.113665 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:37.114153 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:37.114166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.114983 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.114986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:37.115877 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:37.115887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.116311 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.116314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:37.116453 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:37.117033 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:37.117050 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:37.118452 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:37.120167 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:37.120173 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:37.120190 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:37.120531 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:37.120797 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:37.120800 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:37.120802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:37.120975 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:37.121330 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:37.121336 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:37.121590 (+ 254us) client_negotiation.cc:769] Sending connection context
0929 03:15:37.121816 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 03:15:37.122163 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":103,"threads_started":1}
I20260929 03:15:37.124378 18686 heartbeater.cc:347] Connected to a master server at 127.13.238.190:38687
I20260929 03:15:37.124608 18686 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:37.125104 18686 heartbeater.cc:511] Master 127.13.238.190:38687 requested a full tablet report, sending...
I20260929 03:15:37.126179 18363 ts_manager.cc:194] Registered new tserver with Master: 6ce1686e36e046ecb7bc5599a726b928 (127.13.238.130:40529)
WARNING: no policy specified for kudu/127.13.238.131@KRBTEST.COM; defaulting to no policy
I20260929 03:15:37.126881 18363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.130@KRBTEST.COM'} at 127.13.238.130:51811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
I20260929 03:15:37.141234 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:0
--local_ip_for_outbound_sockets=127.13.238.131
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:38687
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:37.253602 18691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:37.253875 18691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:37.253921 18691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:37.257977 18691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:37.258037 18691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:37.258097 18691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:15:37.263439 18691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:38687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:37.264437 18691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:37.265416 18691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:37.274811 18696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:37.274863 18691 server_base.cc:1034] running on GCE node
W20260929 03:15:37.274780 18699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:37.274773 18697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:37.275399 18691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:37.275970 18691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:37.277137 18691 hybrid_clock.cc:648] HybridClock initialized: now 1790651737277100 us; error 47 us; skew 500 ppm
Sep 29 03:15:37 dist-test-slave-nbjg krb5kdc[18319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651737, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:37.279778 18691 init.cc:377] Logged in from keytab as kudu/127.13.238.131@KRBTEST.COM (short username kudu)
I20260929 03:15:37.280465 18691 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:37.280526 18691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:37.280694 18691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:37.282774 18691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-2/data/instance:
uuid: "861e6b70cb544681b5c56fe04c4993bc"
format_stamp: "Formatted at 2026-09-29 03:15:37 on dist-test-slave-nbjg"
server_key: "2bf5833ef3cf513fc0bdcddd5b431988"
server_key_iv: "4fbd122e9ca51095485d8223acad3559"
server_key_version: "encryptionkey@0"
I20260929 03:15:37.283299 18691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance:
uuid: "861e6b70cb544681b5c56fe04c4993bc"
format_stamp: "Formatted at 2026-09-29 03:15:37 on dist-test-slave-nbjg"
server_key: "2bf5833ef3cf513fc0bdcddd5b431988"
server_key_iv: "4fbd122e9ca51095485d8223acad3559"
server_key_version: "encryptionkey@0"
I20260929 03:15:37.287005 18691 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:15:37.289469 18705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:37.290773 18691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 03:15:37.290912 18691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "861e6b70cb544681b5c56fe04c4993bc"
format_stamp: "Formatted at 2026-09-29 03:15:37 on dist-test-slave-nbjg"
server_key: "2bf5833ef3cf513fc0bdcddd5b431988"
server_key_iv: "4fbd122e9ca51095485d8223acad3559"
server_key_version: "encryptionkey@0"
I20260929 03:15:37.291023 18691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:37.306257 18691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:37.309325 18691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:37.309496 18691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:37.310066 18691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:37.310912 18691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:37.310959 18691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:37.311012 18691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:37.311040 18691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:37.319305 18691 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:34863
I20260929 03:15:37.319322 18818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:34863 every 8 connection(s)
I20260929 03:15:37.319886 18691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:15:37.326967 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 18691
I20260929 03:15:37.327068 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance
I20260929 03:15:37.327359 14266 external_mini_cluster.cc:1428] Setting key 01dfa914d9e57b15ea97e7f7716933a2
Sep 29 03:15:37 dist-test-slave-nbjg krb5kdc[18319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651737, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:37.332752 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:37.321593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:53545 (local address 127.13.238.190:38687)
0929 03:15:37.321816 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:37.321821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:37.322662 (+ 841us) server_negotiation.cc:407] Connection header received
0929 03:15:37.323931 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:37.323934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:37.323983 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:37.324065 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:37.325208 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.325747 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.326932 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.327190 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.327280 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:37.329959 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:37.329962 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:37.329964 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:37.329989 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:37.331132 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:37.331570 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:37.331572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:37.331573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:37.331638 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:37.331885 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:37.331887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:37.331888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:37.332047 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:37.332116 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:37.332545 (+ 429us) server_negotiation.cc:299] Negotiation successful
0929 03:15:37.332637 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 03:15:37.333621 18821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:37.322010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:38687 (local address 127.13.238.131:53545)
0929 03:15:37.322495 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:37.322544 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:37.323714 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:37.324217 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:37.324224 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:37.324584 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:37.325059 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:37.325073 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.325839 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.325841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:37.326795 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:37.326806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.327348 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.327352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:37.327572 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:37.328122 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:37.328140 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:37.329820 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:37.331234 (+ 1414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:37.331239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:37.331253 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:37.331496 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:37.331696 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:37.331698 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:37.331699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:37.331825 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:37.332128 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:37.332133 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:37.332360 (+ 227us) client_negotiation.cc:769] Sending connection context
0929 03:15:37.332511 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 03:15:37.332825 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":138,"threads_started":1}
I20260929 03:15:37.334688 18819 heartbeater.cc:347] Connected to a master server at 127.13.238.190:38687
I20260929 03:15:37.334914 18819 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:37.335340 18819 heartbeater.cc:511] Master 127.13.238.190:38687 requested a full tablet report, sending...
I20260929 03:15:37.336338 18363 ts_manager.cc:194] Registered new tserver with Master: 861e6b70cb544681b5c56fe04c4993bc (127.13.238.131:34863)
I20260929 03:15:37.336886 18363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.131@KRBTEST.COM'} at 127.13.238.131:53545
I20260929 03:15:37.341171 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:15:37.349648 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:37.342950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35126 (local address 127.13.238.190:38687)
0929 03:15:37.343081 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:37.343085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:37.343202 (+ 117us) server_negotiation.cc:407] Connection header received
0929 03:15:37.343294 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:37.343296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:37.343333 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:37.343394 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:37.343972 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.344474 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.345308 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.345523 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.345607 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:37.346831 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:37.346834 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:37.346836 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:37.346856 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:37.348262 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:37.348649 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:37.348651 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:37.348652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:37.348726 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:37.348947 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:37.348949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:37.348950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:37.349143 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:37.349241 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:37.349396 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 03:15:37.349482 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 03:15:37.352224 18363 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:15:37.355140 18363 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:15:37.369534 18831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:37.363552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:40529 (local address 127.0.0.1:33900)
0929 03:15:37.364519 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:37.364569 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:37.364696 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:37.365567 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:37.365570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:37.365588 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:37.365690 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:37.365697 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.366771 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.366775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:37.368019 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:37.368028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.368974 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.368978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:37.369098 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:37.369140 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 03:15:37.369246 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 03:15:37.369363 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":25,"thread_start_us":96,"threads_started":1}
I20260929 03:15:37.369601 18830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:37.364178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:36863 (local address 127.0.0.1:56048)
0929 03:15:37.364941 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:37.364951 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:37.365011 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:37.365432 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:37.365439 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:37.365461 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:37.365595 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:37.365603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.367037 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.367040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:37.368229 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:37.368239 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.369190 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.369194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:37.369276 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:37.369297 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 03:15:37.369389 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 03:15:37.369461 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":6,"thread_start_us":84,"threads_started":1}
I20260929 03:15:37.369637 18834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:37.364351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56048 (local address 127.13.238.129:36863)
0929 03:15:37.365033 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:37.365043 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:37.365064 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:15:37.365125 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:37.365129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:37.365287 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:37.365432 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:37.365722 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.366874 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.368346 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.369092 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.369199 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:37.369318 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:37.369417 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 03:15:37.369506 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":109,"threads_started":1}
I20260929 03:15:37.370160 18833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:37.363690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:34863 (local address 127.0.0.1:58718)
0929 03:15:37.364661 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:37.364673 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:37.364739 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:37.366131 (+ 1392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:37.366134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:37.366153 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:37.366251 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:37.366258 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.367563 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.367566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:37.368555 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:37.368563 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.369609 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.369612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:37.369895 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:37.369927 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 03:15:37.369986 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 03:15:37.370046 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":91,"threads_started":1}
I20260929 03:15:37.370491 18832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:37.363690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33900 (local address 127.13.238.130:40529)
0929 03:15:37.365195 (+ 1505us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:37.365203 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:37.365220 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:15:37.365280 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:37.365284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:37.365440 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:37.365571 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:37.365801 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.366663 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.368130 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.368838 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.368975 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:37.369102 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:37.370233 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0929 03:15:37.370338 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1424,"thread_start_us":111,"threads_started":1}
I20260929 03:15:37.370666 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:37.364594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58718 (local address 127.13.238.131:34863)
0929 03:15:37.365727 (+ 1133us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:37.365738 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:37.365757 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:15:37.365823 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:37.365827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:37.365998 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:37.366136 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:37.366367 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.367408 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.368658 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.369332 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.369433 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:37.369547 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:37.370432 (+ 885us) server_negotiation.cc:299] Negotiation successful
0929 03:15:37.370526 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1037,"thread_start_us":151,"threads_started":1}
I20260929 03:15:37.372367 18487 tablet_service.cc:1467] Processing CreateTablet for tablet 5ab95e8fa01140fbbe8ad3072058c064 (DEFAULT_TABLE table=test-table [id=18153f474e0d4f8da7ac02e61cf8191a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:37.373050 18753 tablet_service.cc:1467] Processing CreateTablet for tablet 5ab95e8fa01140fbbe8ad3072058c064 (DEFAULT_TABLE table=test-table [id=18153f474e0d4f8da7ac02e61cf8191a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:37.373425 18487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ab95e8fa01140fbbe8ad3072058c064. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:37.374009 18753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ab95e8fa01140fbbe8ad3072058c064. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:37.375495 18620 tablet_service.cc:1467] Processing CreateTablet for tablet 5ab95e8fa01140fbbe8ad3072058c064 (DEFAULT_TABLE table=test-table [id=18153f474e0d4f8da7ac02e61cf8191a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:37.376464 18620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ab95e8fa01140fbbe8ad3072058c064. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:37.379710 18836 tablet_bootstrap.cc:492] T 5ab95e8fa01140fbbe8ad3072058c064 P 5f7403498aec4818b22305d21cc72c7f: Bootstrap starting.
I20260929 03:15:37.381487 18837 tablet_bootstrap.cc:492] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc: Bootstrap starting.
I20260929 03:15:37.382391 18836 tablet_bootstrap.cc:654] T 5ab95e8fa01140fbbe8ad3072058c064 P 5f7403498aec4818b22305d21cc72c7f: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:37.383114 18838 tablet_bootstrap.cc:492] T 5ab95e8fa01140fbbe8ad3072058c064 P 6ce1686e36e046ecb7bc5599a726b928: Bootstrap starting.
I20260929 03:15:37.383386 18836 log.cc:824] T 5ab95e8fa01140fbbe8ad3072058c064 P 5f7403498aec4818b22305d21cc72c7f: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:37.383739 18837 tablet_bootstrap.cc:654] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:37.384898 18837 log.cc:824] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:37.385135 18838 tablet_bootstrap.cc:654] T 5ab95e8fa01140fbbe8ad3072058c064 P 6ce1686e36e046ecb7bc5599a726b928: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:37.385710 18836 tablet_bootstrap.cc:492] T 5ab95e8fa01140fbbe8ad3072058c064 P 5f7403498aec4818b22305d21cc72c7f: No bootstrap required, opened a new log
I20260929 03:15:37.385959 18836 ts_tablet_manager.cc:1397] T 5ab95e8fa01140fbbe8ad3072058c064 P 5f7403498aec4818b22305d21cc72c7f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 03:15:37.386234 18838 log.cc:824] T 5ab95e8fa01140fbbe8ad3072058c064 P 6ce1686e36e046ecb7bc5599a726b928: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:37.387048 18837 tablet_bootstrap.cc:492] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc: No bootstrap required, opened a new log
I20260929 03:15:37.387269 18837 ts_tablet_manager.cc:1397] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 03:15:37.387809 18838 tablet_bootstrap.cc:492] T 5ab95e8fa01140fbbe8ad3072058c064 P 6ce1686e36e046ecb7bc5599a726b928: No bootstrap required, opened a new log
I20260929 03:15:37.387976 18838 ts_tablet_manager.cc:1397] T 5ab95e8fa01140fbbe8ad3072058c064 P 6ce1686e36e046ecb7bc5599a726b928: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 03:15:37.389959 18836 raft_consensus.cc:348] T 5ab95e8fa01140fbbe8ad3072058c064 P 5f7403498aec4818b22305d21cc72c7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce1686e36e046ecb7bc5599a726b928" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 40529 } } peers { permanent_uuid: "861e6b70cb544681b5c56fe04c4993bc" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 34863 } } peers { permanent_uuid: "5f7403498aec4818b22305d21cc72c7f" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36863 } }
I20260929 03:15:37.390365 18836 raft_consensus.cc:374] T 5ab95e8fa01140fbbe8ad3072058c064 P 5f7403498aec4818b22305d21cc72c7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:37.390465 18836 raft_consensus.cc:729] T 5ab95e8fa01140fbbe8ad3072058c064 P 5f7403498aec4818b22305d21cc72c7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f7403498aec4818b22305d21cc72c7f, State: Initialized, Role: FOLLOWER
I20260929 03:15:37.390949 18836 consensus_queue.cc:260] T 5ab95e8fa01140fbbe8ad3072058c064 P 5f7403498aec4818b22305d21cc72c7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce1686e36e046ecb7bc5599a726b928" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 40529 } } peers { permanent_uuid: "861e6b70cb544681b5c56fe04c4993bc" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 34863 } } peers { permanent_uuid: "5f7403498aec4818b22305d21cc72c7f" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36863 } }
I20260929 03:15:37.390925 18838 raft_consensus.cc:348] T 5ab95e8fa01140fbbe8ad3072058c064 P 6ce1686e36e046ecb7bc5599a726b928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce1686e36e046ecb7bc5599a726b928" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 40529 } } peers { permanent_uuid: "861e6b70cb544681b5c56fe04c4993bc" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 34863 } } peers { permanent_uuid: "5f7403498aec4818b22305d21cc72c7f" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36863 } }
I20260929 03:15:37.390955 18837 raft_consensus.cc:348] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce1686e36e046ecb7bc5599a726b928" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 40529 } } peers { permanent_uuid: "861e6b70cb544681b5c56fe04c4993bc" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 34863 } } peers { permanent_uuid: "5f7403498aec4818b22305d21cc72c7f" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36863 } }
I20260929 03:15:37.391171 18838 raft_consensus.cc:374] T 5ab95e8fa01140fbbe8ad3072058c064 P 6ce1686e36e046ecb7bc5599a726b928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:37.391194 18837 raft_consensus.cc:374] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:37.391218 18838 raft_consensus.cc:729] T 5ab95e8fa01140fbbe8ad3072058c064 P 6ce1686e36e046ecb7bc5599a726b928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ce1686e36e046ecb7bc5599a726b928, State: Initialized, Role: FOLLOWER
I20260929 03:15:37.391245 18837 raft_consensus.cc:729] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861e6b70cb544681b5c56fe04c4993bc, State: Initialized, Role: FOLLOWER
I20260929 03:15:37.391660 18838 consensus_queue.cc:260] T 5ab95e8fa01140fbbe8ad3072058c064 P 6ce1686e36e046ecb7bc5599a726b928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce1686e36e046ecb7bc5599a726b928" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 40529 } } peers { permanent_uuid: "861e6b70cb544681b5c56fe04c4993bc" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 34863 } } peers { permanent_uuid: "5f7403498aec4818b22305d21cc72c7f" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36863 } }
I20260929 03:15:37.391693 18837 consensus_queue.cc:260] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce1686e36e046ecb7bc5599a726b928" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 40529 } } peers { permanent_uuid: "861e6b70cb544681b5c56fe04c4993bc" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 34863 } } peers { permanent_uuid: "5f7403498aec4818b22305d21cc72c7f" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36863 } }
I20260929 03:15:37.392359 18836 ts_tablet_manager.cc:1428] T 5ab95e8fa01140fbbe8ad3072058c064 P 5f7403498aec4818b22305d21cc72c7f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 03:15:37.392428 18819 heartbeater.cc:503] Master 127.13.238.190:38687 was elected leader, sending a full tablet report...
I20260929 03:15:37.392597 18837 ts_tablet_manager.cc:1428] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260929 03:15:37.392650 18838 ts_tablet_manager.cc:1428] T 5ab95e8fa01140fbbe8ad3072058c064 P 6ce1686e36e046ecb7bc5599a726b928: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260929 03:15:37.392740 18553 heartbeater.cc:503] Master 127.13.238.190:38687 was elected leader, sending a full tablet report...
I20260929 03:15:37.393388 18686 heartbeater.cc:503] Master 127.13.238.190:38687 was elected leader, sending a full tablet report...
I20260929 03:15:37.403018 18843 raft_consensus.cc:482] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:15:37.403162 18843 raft_consensus.cc:504] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce1686e36e046ecb7bc5599a726b928" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 40529 } } peers { permanent_uuid: "861e6b70cb544681b5c56fe04c4993bc" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 34863 } } peers { permanent_uuid: "5f7403498aec4818b22305d21cc72c7f" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36863 } }
I20260929 03:15:37.404165 18843 leader_election.cc:288] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ce1686e36e046ecb7bc5599a726b928 (127.13.238.130:40529), 5f7403498aec4818b22305d21cc72c7f (127.13.238.129:36863)
I20260929 03:15:37.408900 18832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:37.404406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:47193 (local address 127.13.238.130:40529)
0929 03:15:37.404612 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:37.404616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:37.404629 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:15:37.404672 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:37.404675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:37.404717 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:37.404786 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:37.405040 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.405820 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.407645 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.408266 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.408384 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:37.408433 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:37.408713 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 03:15:37.408778 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260929 03:15:37.408900 18834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:37.404430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:41057 (local address 127.13.238.129:36863)
0929 03:15:37.404531 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:37.404535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:37.404732 (+ 197us) server_negotiation.cc:407] Connection header received
0929 03:15:37.404831 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:37.404833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:37.404874 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:37.404929 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:37.405144 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.405853 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.407695 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.408266 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.408392 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:37.408433 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:37.408710 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 03:15:37.408778 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 03:15:37.408950 18821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:37.404329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:40529 (local address 127.13.238.131:47193)
0929 03:15:37.404504 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:37.404518 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:37.404598 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:37.404809 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:37.404811 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:37.404826 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:37.404940 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:37.404946 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.405933 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.405936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:37.407531 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:37.407543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.408369 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.408373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:37.408579 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:37.408616 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 03:15:37.408685 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 03:15:37.408779 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":287}
I20260929 03:15:37.408964 18845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:37.404370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:36863 (local address 127.13.238.131:41057)
0929 03:15:37.404650 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:37.404661 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:37.404746 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:37.404946 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:37.404949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:37.404962 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:37.405061 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:37.405066 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.405932 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.405936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:37.407531 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:37.407572 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.408369 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.408373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:37.408582 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:37.408616 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 03:15:37.408689 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 03:15:37.408779 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":168,"thread_start_us":108,"threads_started":1}
I20260929 03:15:37.409554 18507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab95e8fa01140fbbe8ad3072058c064" candidate_uuid: "861e6b70cb544681b5c56fe04c4993bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f7403498aec4818b22305d21cc72c7f" is_pre_election: true
I20260929 03:15:37.409745 18640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab95e8fa01140fbbe8ad3072058c064" candidate_uuid: "861e6b70cb544681b5c56fe04c4993bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ce1686e36e046ecb7bc5599a726b928" is_pre_election: true
I20260929 03:15:37.409873 18507 raft_consensus.cc:2454] T 5ab95e8fa01140fbbe8ad3072058c064 P 5f7403498aec4818b22305d21cc72c7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 861e6b70cb544681b5c56fe04c4993bc in term 0.
I20260929 03:15:37.409977 18640 raft_consensus.cc:2454] T 5ab95e8fa01140fbbe8ad3072058c064 P 6ce1686e36e046ecb7bc5599a726b928 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 861e6b70cb544681b5c56fe04c4993bc in term 0.
I20260929 03:15:37.410413 18706 leader_election.cc:302] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f7403498aec4818b22305d21cc72c7f, 861e6b70cb544681b5c56fe04c4993bc; no voters:
I20260929 03:15:37.410665 18843 raft_consensus.cc:2781] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:15:37.410740 18843 raft_consensus.cc:482] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:15:37.410789 18843 raft_consensus.cc:3037] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc [term 0 FOLLOWER]: Advancing to term 1
W20260929 03:15:37.411696 18554 tablet.cc:2369] T 5ab95e8fa01140fbbe8ad3072058c064 P 5f7403498aec4818b22305d21cc72c7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:15:37.411808 18843 raft_consensus.cc:504] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce1686e36e046ecb7bc5599a726b928" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 40529 } } peers { permanent_uuid: "861e6b70cb544681b5c56fe04c4993bc" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 34863 } } peers { permanent_uuid: "5f7403498aec4818b22305d21cc72c7f" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36863 } }
I20260929 03:15:37.412119 18843 leader_election.cc:288] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc [CANDIDATE]: Term 1 election: Requested vote from peers 6ce1686e36e046ecb7bc5599a726b928 (127.13.238.130:40529), 5f7403498aec4818b22305d21cc72c7f (127.13.238.129:36863)
I20260929 03:15:37.412442 18640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab95e8fa01140fbbe8ad3072058c064" candidate_uuid: "861e6b70cb544681b5c56fe04c4993bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ce1686e36e046ecb7bc5599a726b928"
I20260929 03:15:37.412474 18507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab95e8fa01140fbbe8ad3072058c064" candidate_uuid: "861e6b70cb544681b5c56fe04c4993bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f7403498aec4818b22305d21cc72c7f"
I20260929 03:15:37.412561 18640 raft_consensus.cc:3037] T 5ab95e8fa01140fbbe8ad3072058c064 P 6ce1686e36e046ecb7bc5599a726b928 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:37.412568 18507 raft_consensus.cc:3037] T 5ab95e8fa01140fbbe8ad3072058c064 P 5f7403498aec4818b22305d21cc72c7f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:37.413579 18507 raft_consensus.cc:2454] T 5ab95e8fa01140fbbe8ad3072058c064 P 5f7403498aec4818b22305d21cc72c7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 861e6b70cb544681b5c56fe04c4993bc in term 1.
I20260929 03:15:37.413600 18640 raft_consensus.cc:2454] T 5ab95e8fa01140fbbe8ad3072058c064 P 6ce1686e36e046ecb7bc5599a726b928 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 861e6b70cb544681b5c56fe04c4993bc in term 1.
I20260929 03:15:37.413839 18706 leader_election.cc:302] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f7403498aec4818b22305d21cc72c7f, 861e6b70cb544681b5c56fe04c4993bc; no voters:
I20260929 03:15:37.413996 18843 raft_consensus.cc:2781] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:37.414247 18843 raft_consensus.cc:686] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc [term 1 LEADER]: Becoming Leader. State: Replica: 861e6b70cb544681b5c56fe04c4993bc, State: Running, Role: LEADER
I20260929 03:15:37.414595 18843 consensus_queue.cc:237] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce1686e36e046ecb7bc5599a726b928" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 40529 } } peers { permanent_uuid: "861e6b70cb544681b5c56fe04c4993bc" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 34863 } } peers { permanent_uuid: "5f7403498aec4818b22305d21cc72c7f" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36863 } }
I20260929 03:15:37.417609 18362 catalog_manager.cc:5495] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 861e6b70cb544681b5c56fe04c4993bc (127.13.238.131). New cstate: current_term: 1 leader_uuid: "861e6b70cb544681b5c56fe04c4993bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce1686e36e046ecb7bc5599a726b928" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 40529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861e6b70cb544681b5c56fe04c4993bc" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f7403498aec4818b22305d21cc72c7f" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36863 } health_report { overall_health: UNKNOWN } } }
I20260929 03:15:37.430977 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:37.427478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58726 (local address 127.13.238.131:34863)
0929 03:15:37.427591 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:37.427595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:37.427609 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:15:37.427778 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:37.427781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:37.427834 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:37.427939 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:37.428169 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.428667 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.429500 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:37.429709 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:37.429795 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:37.430090 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:37.430576 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:37.430691 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:37.430762 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 03:15:37.430824 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 03:15:37.436936 18640 raft_consensus.cc:1261] T 5ab95e8fa01140fbbe8ad3072058c064 P 6ce1686e36e046ecb7bc5599a726b928 [term 1 FOLLOWER]: Refusing update from remote peer 861e6b70cb544681b5c56fe04c4993bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:37.436949 18507 raft_consensus.cc:1261] T 5ab95e8fa01140fbbe8ad3072058c064 P 5f7403498aec4818b22305d21cc72c7f [term 1 FOLLOWER]: Refusing update from remote peer 861e6b70cb544681b5c56fe04c4993bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:37.437584 18846 consensus_queue.cc:1035] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ce1686e36e046ecb7bc5599a726b928" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 40529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:37.437772 18843 consensus_queue.cc:1035] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f7403498aec4818b22305d21cc72c7f" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:37.447943 18850 mvcc.cc:205] Tried to move back new op lower bound from 7334509516534579200 to 7334509516452823040. Current Snapshot: MvccSnapshot[applied={T|T < 7334509516534579200 or (T in {7334509516534579200})}]
I20260929 03:15:37.449131 18849 mvcc.cc:205] Tried to move back new op lower bound from 7334509516534579200 to 7334509516452823040. Current Snapshot: MvccSnapshot[applied={T|T < 7334509516534579200 or (T in {7334509516534579200})}]
I20260929 03:15:37.456664 18362 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:37.456826 18362 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:37.459256 18362 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64a3726e22be48ab8258c9d22f668124: Sending DeleteTablet for 3 replicas of tablet 5ab95e8fa01140fbbe8ad3072058c064
I20260929 03:15:37.460044 18620 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ab95e8fa01140fbbe8ad3072058c064 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:37 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:33900
I20260929 03:15:37.460183 18487 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ab95e8fa01140fbbe8ad3072058c064 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:37 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:56048
I20260929 03:15:37.460345 18753 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ab95e8fa01140fbbe8ad3072058c064 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:37 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:58718
I20260929 03:15:37.460762 18859 tablet_replica.cc:330] T 5ab95e8fa01140fbbe8ad3072058c064 P 6ce1686e36e046ecb7bc5599a726b928: stopping tablet replica
I20260929 03:15:37.460767 18861 tablet_replica.cc:330] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc: stopping tablet replica
I20260929 03:15:37.461193 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 18425
I20260929 03:15:37.461205 18861 raft_consensus.cc:2229] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:15:37.462332 18859 raft_consensus.cc:2229] T 5ab95e8fa01140fbbe8ad3072058c064 P 6ce1686e36e046ecb7bc5599a726b928 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:15:37.462778 18859 raft_consensus.cc:2258] T 5ab95e8fa01140fbbe8ad3072058c064 P 6ce1686e36e046ecb7bc5599a726b928 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:37.464226 18861 raft_consensus.cc:2258] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:37.464766 18859 ts_tablet_manager.cc:1905] T 5ab95e8fa01140fbbe8ad3072058c064 P 6ce1686e36e046ecb7bc5599a726b928: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:37.465757 18861 ts_tablet_manager.cc:1905] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:37.467814 18859 ts_tablet_manager.cc:1918] T 5ab95e8fa01140fbbe8ad3072058c064 P 6ce1686e36e046ecb7bc5599a726b928: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:15:37.467950 18859 log.cc:1196] T 5ab95e8fa01140fbbe8ad3072058c064 P 6ce1686e36e046ecb7bc5599a726b928: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-1/wal/wals/5ab95e8fa01140fbbe8ad3072058c064
I20260929 03:15:37.468277 18859 ts_tablet_manager.cc:1939] T 5ab95e8fa01140fbbe8ad3072058c064 P 6ce1686e36e046ecb7bc5599a726b928: Deleting consensus metadata
I20260929 03:15:37.468575 18861 ts_tablet_manager.cc:1918] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:15:37.468729 18861 log.cc:1196] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/5ab95e8fa01140fbbe8ad3072058c064
W20260929 03:15:37.468751 18347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.238.129:36863 (error 108)
I20260929 03:15:37.469133 18861 ts_tablet_manager.cc:1939] T 5ab95e8fa01140fbbe8ad3072058c064 P 861e6b70cb544681b5c56fe04c4993bc: Deleting consensus metadata
I20260929 03:15:37.469576 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 18558
I20260929 03:15:37.470762 18349 catalog_manager.cc:4841] TS 861e6b70cb544681b5c56fe04c4993bc (127.13.238.131:34863): tablet 5ab95e8fa01140fbbe8ad3072058c064 (table test-table [id=18153f474e0d4f8da7ac02e61cf8191a]) successfully deleted
I20260929 03:15:37.475490 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 18691
I20260929 03:15:37.476712 18833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:37.476398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:36863 (local address 127.0.0.1:56062)
0929 03:15:37.476539 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:37.476618 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.129:36863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 03:15:37.476852 18830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:37.476398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:40529 (local address 127.0.0.1:33916)
0929 03:15:37.476711 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:37.476760 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.130:40529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":192}
W20260929 03:15:37.476931 18347 catalog_manager.cc:4584] TS 5f7403498aec4818b22305d21cc72c7f (127.13.238.129:36863): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5ab95e8fa01140fbbe8ad3072058c064: Network error: Client connection negotiation failed: client connection to 127.13.238.129:36863: connect: Connection refused (error 111)
I20260929 03:15:37.480931 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 18332
2026-09-29T03:15:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2102 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T03:15:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:15:37Z Disabled control of system clock
I20260929 03:15:37.508754 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:43131
--webserver_interface=127.13.238.190
--webserver_port=0
--builtin_ntp_servers=127.13.238.148:34559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:43131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 03:15:37.617520 18866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:37.617803 18866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:37.617844 18866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:37.621816 18866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:37.621876 18866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:37.621889 18866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:37.621905 18866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:37.621932 18866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:15:37.626817 18866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:43131
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:43131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:37.627904 18866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:37.629323 18866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:37.637413 18872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:37.637532 18866 server_base.cc:1034] running on GCE node
W20260929 03:15:37.637449 18871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:37.637413 18874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:37.638324 18866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:37.639357 18866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:37.640517 18866 hybrid_clock.cc:648] HybridClock initialized: now 1790651737640497 us; error 31 us; skew 500 ppm
I20260929 03:15:37.642395 18866 webserver.cc:460] Webserver started at http://127.13.238.190:36257/ using document root <none> and password file <none>
I20260929 03:15:37.642957 18866 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:37.643023 18866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:37.643210 18866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:37.645234 18866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "1013eee3bbd94d93956a627be5ac5f06"
format_stamp: "Formatted at 2026-09-29 03:15:37 on dist-test-slave-nbjg"
server_key: "2addaf2594cd262833915df15cca19e2"
server_key_iv: "45cb7c60cd25a15f2d6cbd310e9473c7"
server_key_version: "encryptionkey@0"
I20260929 03:15:37.645766 18866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "1013eee3bbd94d93956a627be5ac5f06"
format_stamp: "Formatted at 2026-09-29 03:15:37 on dist-test-slave-nbjg"
server_key: "2addaf2594cd262833915df15cca19e2"
server_key_iv: "45cb7c60cd25a15f2d6cbd310e9473c7"
server_key_version: "encryptionkey@0"
I20260929 03:15:37.649384 18866 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260929 03:15:37.651890 18880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:37.653110 18866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 03:15:37.653220 18866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "1013eee3bbd94d93956a627be5ac5f06"
format_stamp: "Formatted at 2026-09-29 03:15:37 on dist-test-slave-nbjg"
server_key: "2addaf2594cd262833915df15cca19e2"
server_key_iv: "45cb7c60cd25a15f2d6cbd310e9473c7"
server_key_version: "encryptionkey@0"
I20260929 03:15:37.653330 18866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:37.657685 18866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:37.658296 18866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:37.658447 18866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:37.664834 18866 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:43131
I20260929 03:15:37.664846 18932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:43131 every 8 connection(s)
I20260929 03:15:37.665704 18866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:15:37.668809 18933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:37.673978 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 18866
I20260929 03:15:37.674103 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:15:37.674446 14266 external_mini_cluster.cc:1428] Setting key 00f7850fbee70c0219bb77db76e033c8
I20260929 03:15:37.674515 18933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1013eee3bbd94d93956a627be5ac5f06: Bootstrap starting.
I20260929 03:15:37.677052 18933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1013eee3bbd94d93956a627be5ac5f06: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:37.677783 18933 log.cc:824] T 00000000000000000000000000000000 P 1013eee3bbd94d93956a627be5ac5f06: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:37.679805 18933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1013eee3bbd94d93956a627be5ac5f06: No bootstrap required, opened a new log
I20260929 03:15:37.680058 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:37.675878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52352 (local address 127.13.238.190:43131)
0929 03:15:37.676529 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:37.676544 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:37.676578 (+ 34us) server_negotiation.cc:407] Connection header received
0929 03:15:37.677284 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:37.677304 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:37.677658 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:37.677989 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:37.678780 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:37.678785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:37.678797 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:15:37.678807 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:37.678860 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:15:37.678875 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:37.678956 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:37.679068 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 03:15:37.679311 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":122,"threads_started":1}
I20260929 03:15:37.683553 18933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1013eee3bbd94d93956a627be5ac5f06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1013eee3bbd94d93956a627be5ac5f06" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43131 } }
I20260929 03:15:37.683759 18933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1013eee3bbd94d93956a627be5ac5f06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:37.683799 18933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1013eee3bbd94d93956a627be5ac5f06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1013eee3bbd94d93956a627be5ac5f06, State: Initialized, Role: FOLLOWER
I20260929 03:15:37.684208 18933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1013eee3bbd94d93956a627be5ac5f06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1013eee3bbd94d93956a627be5ac5f06" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43131 } }
I20260929 03:15:37.684329 18933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1013eee3bbd94d93956a627be5ac5f06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:15:37.684387 18933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1013eee3bbd94d93956a627be5ac5f06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:15:37.684484 18933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1013eee3bbd94d93956a627be5ac5f06 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:37.685281 18933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1013eee3bbd94d93956a627be5ac5f06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1013eee3bbd94d93956a627be5ac5f06" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43131 } }
I20260929 03:15:37.685598 18933 leader_election.cc:302] T 00000000000000000000000000000000 P 1013eee3bbd94d93956a627be5ac5f06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1013eee3bbd94d93956a627be5ac5f06; no voters:
I20260929 03:15:37.685880 18933 leader_election.cc:288] T 00000000000000000000000000000000 P 1013eee3bbd94d93956a627be5ac5f06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:15:37.685995 18938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1013eee3bbd94d93956a627be5ac5f06 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:37.686187 18938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1013eee3bbd94d93956a627be5ac5f06 [term 1 LEADER]: Becoming Leader. State: Replica: 1013eee3bbd94d93956a627be5ac5f06, State: Running, Role: LEADER
I20260929 03:15:37.686542 18938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1013eee3bbd94d93956a627be5ac5f06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1013eee3bbd94d93956a627be5ac5f06" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43131 } }
I20260929 03:15:37.687043 18933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1013eee3bbd94d93956a627be5ac5f06 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:37.688120 18940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1013eee3bbd94d93956a627be5ac5f06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1013eee3bbd94d93956a627be5ac5f06. Latest consensus state: current_term: 1 leader_uuid: "1013eee3bbd94d93956a627be5ac5f06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1013eee3bbd94d93956a627be5ac5f06" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43131 } } }
I20260929 03:15:37.688242 18940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1013eee3bbd94d93956a627be5ac5f06 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:37.688433 18939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1013eee3bbd94d93956a627be5ac5f06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1013eee3bbd94d93956a627be5ac5f06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1013eee3bbd94d93956a627be5ac5f06" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43131 } } }
I20260929 03:15:37.688529 18939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1013eee3bbd94d93956a627be5ac5f06 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:37.688782 18947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:15:37.691676 18947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:15:37.697245 18947 catalog_manager.cc:1347] Generated new cluster ID: e30431597f0c47d7aaa814ac170f3408
I20260929 03:15:37.697314 18947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:15:37.718139 18947 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:15:37.718569 18947 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:15:37.725031 18947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1013eee3bbd94d93956a627be5ac5f06: Generated new TSK 0
I20260929 03:15:37.725703 18947 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:15:37.730408 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:0
--local_ip_for_outbound_sockets=127.13.238.129
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43131
--builtin_ntp_servers=127.13.238.148:34559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 03:15:37.856600 18957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:37.856846 18957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:37.856884 18957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:37.860872 18957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:37.860924 18957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:37.860978 18957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:15:37.865628 18957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:37.866709 18957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:37.868050 18957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:37.877400 18963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:37.877453 18962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:37.877425 18965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:37.877609 18957 server_base.cc:1034] running on GCE node
I20260929 03:15:37.878029 18957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:37.878676 18957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:37.879868 18957 hybrid_clock.cc:648] HybridClock initialized: now 1790651737879840 us; error 37 us; skew 500 ppm
I20260929 03:15:37.881659 18957 webserver.cc:460] Webserver started at http://127.13.238.129:40825/ using document root <none> and password file <none>
I20260929 03:15:37.882253 18957 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:37.882318 18957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:37.882516 18957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:37.884521 18957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-0/data/instance:
uuid: "3e02b631e1fb4df097767a3ad243f0f7"
format_stamp: "Formatted at 2026-09-29 03:15:37 on dist-test-slave-nbjg"
server_key: "4d51f818feba04a94f7442f05670669d"
server_key_iv: "2734ee8c7f6ec0916c0f2c7e02494164"
server_key_version: "encryptionkey@0"
I20260929 03:15:37.885043 18957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance:
uuid: "3e02b631e1fb4df097767a3ad243f0f7"
format_stamp: "Formatted at 2026-09-29 03:15:37 on dist-test-slave-nbjg"
server_key: "4d51f818feba04a94f7442f05670669d"
server_key_iv: "2734ee8c7f6ec0916c0f2c7e02494164"
server_key_version: "encryptionkey@0"
I20260929 03:15:37.888716 18957 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 03:15:37.891151 18971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:37.892323 18957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:37.892419 18957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "3e02b631e1fb4df097767a3ad243f0f7"
format_stamp: "Formatted at 2026-09-29 03:15:37 on dist-test-slave-nbjg"
server_key: "4d51f818feba04a94f7442f05670669d"
server_key_iv: "2734ee8c7f6ec0916c0f2c7e02494164"
server_key_version: "encryptionkey@0"
I20260929 03:15:37.892517 18957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:37.896850 18957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:37.897411 18957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:37.897555 18957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:37.898068 18957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:37.898937 18957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:37.898985 18957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:37.899024 18957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:37.899053 18957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:37.906970 18957 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:44637
I20260929 03:15:37.907006 19084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:44637 every 8 connection(s)
I20260929 03:15:37.907814 18957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:15:37.913957 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:37.909561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:49335 (local address 127.13.238.190:43131)
0929 03:15:37.909693 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:37.909696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:37.911011 (+ 1315us) server_negotiation.cc:407] Connection header received
0929 03:15:37.912309 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:37.912313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:37.912361 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:37.912442 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:37.913506 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:37.913511 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:37.913514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:15:37.913529 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:37.913550 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:15:37.913557 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:37.913636 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:37.913758 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 03:15:37.913806 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:15:37.914839 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:37.910084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43131 (local address 127.13.238.129:49335)
0929 03:15:37.910853 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:37.910895 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:37.912087 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:37.912631 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:37.912638 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:37.912735 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:37.913232 (+ 497us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:15:37.913234 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:37.913276 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:15:37.913357 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:37.913626 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:37.913635 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:37.913640 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:15:37.913798 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 03:15:37.914045 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":141,"threads_started":1}
I20260929 03:15:37.915993 19085 heartbeater.cc:347] Connected to a master server at 127.13.238.190:43131
I20260929 03:15:37.916194 19085 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:37.916558 19085 heartbeater.cc:511] Master 127.13.238.190:43131 requested a full tablet report, sending...
I20260929 03:15:37.917094 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 18957
I20260929 03:15:37.917212 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance
I20260929 03:15:37.917477 14266 external_mini_cluster.cc:1428] Setting key 677bd232d4902e83655e68da7c5a4cb7
I20260929 03:15:37.918126 18897 ts_manager.cc:194] Registered new tserver with Master: 3e02b631e1fb4df097767a3ad243f0f7 (127.13.238.129:44637)
I20260929 03:15:37.919534 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:0
--local_ip_for_outbound_sockets=127.13.238.130
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43131
--builtin_ntp_servers=127.13.238.148:34559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 03:15:38.026989 19088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:38.027242 19088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:38.027281 19088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:38.031215 19088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:38.031265 19088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:38.031320 19088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:15:38.036078 19088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:38.037146 19088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:38.038606 19088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:38.048424 19096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:38.048418 19094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:38.048599 19088 server_base.cc:1034] running on GCE node
W20260929 03:15:38.048418 19093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:38.049235 19088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:38.050050 19088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:38.051249 19088 hybrid_clock.cc:648] HybridClock initialized: now 1790651738051218 us; error 49 us; skew 500 ppm
I20260929 03:15:38.053298 19088 webserver.cc:460] Webserver started at http://127.13.238.130:39511/ using document root <none> and password file <none>
I20260929 03:15:38.053915 19088 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:38.053987 19088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:38.054185 19088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:38.056448 19088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-1/data/instance:
uuid: "aa57c3482e0c4ac3abfcbe2ff72bcacd"
format_stamp: "Formatted at 2026-09-29 03:15:38 on dist-test-slave-nbjg"
server_key: "ebb1b320e23653fd2cda9089eda607a6"
server_key_iv: "e1a572cc3fae4cb49f3ecaed1ad6e45f"
server_key_version: "encryptionkey@0"
I20260929 03:15:38.057055 19088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance:
uuid: "aa57c3482e0c4ac3abfcbe2ff72bcacd"
format_stamp: "Formatted at 2026-09-29 03:15:38 on dist-test-slave-nbjg"
server_key: "ebb1b320e23653fd2cda9089eda607a6"
server_key_iv: "e1a572cc3fae4cb49f3ecaed1ad6e45f"
server_key_version: "encryptionkey@0"
I20260929 03:15:38.060984 19088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 03:15:38.063643 19102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:38.064886 19088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 03:15:38.065001 19088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "aa57c3482e0c4ac3abfcbe2ff72bcacd"
format_stamp: "Formatted at 2026-09-29 03:15:38 on dist-test-slave-nbjg"
server_key: "ebb1b320e23653fd2cda9089eda607a6"
server_key_iv: "e1a572cc3fae4cb49f3ecaed1ad6e45f"
server_key_version: "encryptionkey@0"
I20260929 03:15:38.065109 19088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:38.069573 19088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:38.070174 19088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:38.070370 19088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:38.070899 19088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:38.071800 19088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:38.071846 19088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:38.071894 19088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:38.071923 19088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:38.079769 19088 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:43375
I20260929 03:15:38.079841 19215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:43375 every 8 connection(s)
I20260929 03:15:38.080617 19088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:15:38.085206 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 19088
I20260929 03:15:38.085325 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance
I20260929 03:15:38.085579 14266 external_mini_cluster.cc:1428] Setting key c19b990ac81c79d706f0baa3c78c2d8c
I20260929 03:15:38.086704 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:38.082421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:55593 (local address 127.13.238.190:43131)
0929 03:15:38.082552 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:38.082555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:38.083662 (+ 1107us) server_negotiation.cc:407] Connection header received
0929 03:15:38.084950 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:38.084953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:38.085002 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:38.085099 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:38.086247 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:38.086251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:38.086254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:15:38.086268 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:38.086292 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:15:38.086302 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:38.086393 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:38.086546 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 03:15:38.086584 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 03:15:38.087569 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:0
--local_ip_for_outbound_sockets=127.13.238.131
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43131
--builtin_ntp_servers=127.13.238.148:34559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 03:15:38.087756 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:38.082887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43131 (local address 127.13.238.130:55593)
0929 03:15:38.083541 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:38.083581 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:38.084732 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:38.085233 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:38.085240 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:38.085348 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:38.085898 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:15:38.085900 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:38.085966 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:15:38.086055 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:38.086398 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:38.086414 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:38.086420 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:15:38.086591 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 03:15:38.086880 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":122,"threads_started":1}
I20260929 03:15:38.088953 19216 heartbeater.cc:347] Connected to a master server at 127.13.238.190:43131
I20260929 03:15:38.089195 19216 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:38.089671 19216 heartbeater.cc:511] Master 127.13.238.190:43131 requested a full tablet report, sending...
I20260929 03:15:38.090788 18897 ts_manager.cc:194] Registered new tserver with Master: aa57c3482e0c4ac3abfcbe2ff72bcacd (127.13.238.130:43375)
W20260929 03:15:38.199481 19219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:38.199740 19219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:38.199779 19219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:38.204171 19219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:38.204236 19219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:38.204313 19219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:15:38.209232 19219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:38.210362 19219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:38.211812 19219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:15:38.221242 19219 server_base.cc:1034] running on GCE node
W20260929 03:15:38.221200 19224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:38.221196 19225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:38.221196 19227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:38.221796 19219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:38.222446 19219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:38.223594 19219 hybrid_clock.cc:648] HybridClock initialized: now 1790651738223572 us; error 32 us; skew 500 ppm
I20260929 03:15:38.225374 19219 webserver.cc:460] Webserver started at http://127.13.238.131:45969/ using document root <none> and password file <none>
I20260929 03:15:38.225934 19219 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:38.225996 19219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:38.226239 19219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:38.228328 19219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-2/data/instance:
uuid: "e2ce84da4de540798b52c01cf4e2161a"
format_stamp: "Formatted at 2026-09-29 03:15:38 on dist-test-slave-nbjg"
server_key: "406ae90938d7e6029ce643c617692f17"
server_key_iv: "97ae3b0835c13efffa555106f03e6731"
server_key_version: "encryptionkey@0"
I20260929 03:15:38.228883 19219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance:
uuid: "e2ce84da4de540798b52c01cf4e2161a"
format_stamp: "Formatted at 2026-09-29 03:15:38 on dist-test-slave-nbjg"
server_key: "406ae90938d7e6029ce643c617692f17"
server_key_iv: "97ae3b0835c13efffa555106f03e6731"
server_key_version: "encryptionkey@0"
I20260929 03:15:38.232839 19219 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:15:38.235563 19233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:38.236856 19219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 03:15:38.236980 19219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "e2ce84da4de540798b52c01cf4e2161a"
format_stamp: "Formatted at 2026-09-29 03:15:38 on dist-test-slave-nbjg"
server_key: "406ae90938d7e6029ce643c617692f17"
server_key_iv: "97ae3b0835c13efffa555106f03e6731"
server_key_version: "encryptionkey@0"
I20260929 03:15:38.237097 19219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:38.241825 19219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:38.242522 19219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:38.242681 19219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:38.243253 19219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:38.244211 19219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:38.244258 19219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:38.244302 19219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:38.244330 19219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:38.252918 19219 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:38583
I20260929 03:15:38.252933 19346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:38583 every 8 connection(s)
I20260929 03:15:38.253878 19219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:15:38.260188 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:38.255772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:52033 (local address 127.13.238.190:43131)
0929 03:15:38.255920 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:38.255924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:38.257105 (+ 1181us) server_negotiation.cc:407] Connection header received
0929 03:15:38.258470 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:38.258474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:38.258524 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:38.258602 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:38.259722 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:38.259726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:38.259728 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:15:38.259742 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:38.259763 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:15:38.259772 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:38.259863 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:38.259993 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 03:15:38.260043 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:15:38.261088 19349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:38.256237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43131 (local address 127.13.238.131:52033)
0929 03:15:38.256932 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:38.256973 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:38.258234 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:38.258796 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:38.258804 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:38.258908 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:38.259437 (+ 529us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:15:38.259439 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:38.259483 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:15:38.259560 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:38.259854 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:38.259862 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:38.259868 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:15:38.260036 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 03:15:38.260315 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":120,"threads_started":1}
I20260929 03:15:38.262220 19347 heartbeater.cc:347] Connected to a master server at 127.13.238.190:43131
I20260929 03:15:38.262444 19347 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:38.262844 19347 heartbeater.cc:511] Master 127.13.238.190:43131 requested a full tablet report, sending...
I20260929 03:15:38.263883 18897 ts_manager.cc:194] Registered new tserver with Master: e2ce84da4de540798b52c01cf4e2161a (127.13.238.131:38583)
I20260929 03:15:38.264092 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 19219
I20260929 03:15:38.264191 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance
I20260929 03:15:38.264495 14266 external_mini_cluster.cc:1428] Setting key 6a40c32312fdcc28b6cc69ec3d43053d
I20260929 03:15:38.266558 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:15:38.270007 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:38.268421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52360 (local address 127.13.238.190:43131)
0929 03:15:38.268550 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:38.268555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:38.268735 (+ 180us) server_negotiation.cc:407] Connection header received
0929 03:15:38.268906 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:38.268909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:38.268945 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:38.269008 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:38.269635 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:38.269639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:38.269641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:15:38.269654 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:38.269680 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:15:38.269688 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:38.269759 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:38.269834 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 03:15:38.269876 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:15:38.272670 18897 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:15:38.275771 18897 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:15:38.287973 19358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:38.284484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:38583 (local address 127.0.0.1:38496)
0929 03:15:38.284903 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:38.284963 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:38.285062 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:38.286312 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:38.286315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:38.286395 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:38.287231 (+ 836us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:15:38.287232 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:38.287239 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:15:38.287265 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:38.287605 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:38.287613 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:38.287620 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:15:38.287730 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 03:15:38.287803 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":71,"threads_started":1}
I20260929 03:15:38.288017 19360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:38.284596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38496 (local address 127.13.238.131:38583)
0929 03:15:38.285827 (+ 1231us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:38.285840 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:38.285865 (+ 25us) server_negotiation.cc:407] Connection header received
0929 03:15:38.285928 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:38.285933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:38.286134 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:38.286692 (+ 558us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:38.287388 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:38.287394 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:38.287407 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:15:38.287424 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:38.287468 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:15:38.287494 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:38.287638 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:38.287765 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 03:15:38.287847 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1123,"thread_start_us":135,"threads_started":1}
I20260929 03:15:38.288292 19363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:38.285352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33920 (local address 127.13.238.130:43375)
0929 03:15:38.285996 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:38.286005 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:38.286025 (+ 20us) server_negotiation.cc:407] Connection header received
0929 03:15:38.286066 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:38.286070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:38.286237 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:38.286693 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:38.287554 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:38.287560 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:38.287572 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:15:38.287584 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:38.287627 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:15:38.287665 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:38.287757 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:38.288042 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 03:15:38.288114 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":82,"threads_started":1}
I20260929 03:15:38.288349 19362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:38.285254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44438 (local address 127.13.238.129:44637)
0929 03:15:38.285832 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:38.285842 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:38.285865 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:15:38.285918 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:38.285924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:38.286134 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:38.286307 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:38.287598 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:38.287603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:38.287614 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:15:38.287624 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:38.287663 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:15:38.287688 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:38.287785 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:38.288043 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 03:15:38.288117 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":129,"threads_started":1}
I20260929 03:15:38.288666 19359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:38.284593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:44637 (local address 127.0.0.1:44438)
0929 03:15:38.285499 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:38.285513 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:38.285585 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:38.286303 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:38.286310 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:38.286395 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:38.287125 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:15:38.287127 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:38.287147 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:15:38.287515 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:38.287828 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:38.287831 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:38.287835 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:15:38.287932 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 03:15:38.287965 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"thread_start_us":88,"threads_started":1}
I20260929 03:15:38.287981 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:38.285193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:43375 (local address 127.0.0.1:33920)
0929 03:15:38.285642 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:38.285653 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:38.285731 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:38.286366 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:38.286369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:38.286394 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:38.287421 (+ 1027us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:15:38.287423 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:38.287430 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:15:38.287447 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:38.287803 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:38.287806 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:38.287809 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:15:38.287872 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 03:15:38.287898 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":91,"threads_started":1}
I20260929 03:15:38.291601 19019 tablet_service.cc:1467] Processing CreateTablet for tablet b722f8ed17134c7aa715550c54a88850 (DEFAULT_TABLE table=test-table [id=774abd3036344fdfabfbafe8a4fb00e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:38.291605 19281 tablet_service.cc:1467] Processing CreateTablet for tablet b722f8ed17134c7aa715550c54a88850 (DEFAULT_TABLE table=test-table [id=774abd3036344fdfabfbafe8a4fb00e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:38.291694 19150 tablet_service.cc:1467] Processing CreateTablet for tablet b722f8ed17134c7aa715550c54a88850 (DEFAULT_TABLE table=test-table [id=774abd3036344fdfabfbafe8a4fb00e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:38.292737 19150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b722f8ed17134c7aa715550c54a88850. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:38.292742 19019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b722f8ed17134c7aa715550c54a88850. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:38.292742 19281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b722f8ed17134c7aa715550c54a88850. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:38.299147 19364 tablet_bootstrap.cc:492] T b722f8ed17134c7aa715550c54a88850 P aa57c3482e0c4ac3abfcbe2ff72bcacd: Bootstrap starting.
I20260929 03:15:38.299799 19365 tablet_bootstrap.cc:492] T b722f8ed17134c7aa715550c54a88850 P 3e02b631e1fb4df097767a3ad243f0f7: Bootstrap starting.
I20260929 03:15:38.302078 19365 tablet_bootstrap.cc:654] T b722f8ed17134c7aa715550c54a88850 P 3e02b631e1fb4df097767a3ad243f0f7: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:38.302472 19364 tablet_bootstrap.cc:654] T b722f8ed17134c7aa715550c54a88850 P aa57c3482e0c4ac3abfcbe2ff72bcacd: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:38.302476 19366 tablet_bootstrap.cc:492] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a: Bootstrap starting.
I20260929 03:15:38.302989 19365 log.cc:824] T b722f8ed17134c7aa715550c54a88850 P 3e02b631e1fb4df097767a3ad243f0f7: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:38.303228 19364 log.cc:824] T b722f8ed17134c7aa715550c54a88850 P aa57c3482e0c4ac3abfcbe2ff72bcacd: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:38.304325 19366 tablet_bootstrap.cc:654] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:38.304853 19365 tablet_bootstrap.cc:492] T b722f8ed17134c7aa715550c54a88850 P 3e02b631e1fb4df097767a3ad243f0f7: No bootstrap required, opened a new log
I20260929 03:15:38.305027 19365 ts_tablet_manager.cc:1397] T b722f8ed17134c7aa715550c54a88850 P 3e02b631e1fb4df097767a3ad243f0f7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 03:15:38.305099 19364 tablet_bootstrap.cc:492] T b722f8ed17134c7aa715550c54a88850 P aa57c3482e0c4ac3abfcbe2ff72bcacd: No bootstrap required, opened a new log
I20260929 03:15:38.305101 19366 log.cc:824] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:38.305281 19364 ts_tablet_manager.cc:1397] T b722f8ed17134c7aa715550c54a88850 P aa57c3482e0c4ac3abfcbe2ff72bcacd: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 03:15:38.306941 19366 tablet_bootstrap.cc:492] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a: No bootstrap required, opened a new log
I20260929 03:15:38.307116 19366 ts_tablet_manager.cc:1397] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 03:15:38.308701 19365 raft_consensus.cc:348] T b722f8ed17134c7aa715550c54a88850 P 3e02b631e1fb4df097767a3ad243f0f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ce84da4de540798b52c01cf4e2161a" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38583 } } peers { permanent_uuid: "3e02b631e1fb4df097767a3ad243f0f7" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 44637 } } peers { permanent_uuid: "aa57c3482e0c4ac3abfcbe2ff72bcacd" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43375 } }
I20260929 03:15:38.308909 19365 raft_consensus.cc:374] T b722f8ed17134c7aa715550c54a88850 P 3e02b631e1fb4df097767a3ad243f0f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:38.308959 19365 raft_consensus.cc:729] T b722f8ed17134c7aa715550c54a88850 P 3e02b631e1fb4df097767a3ad243f0f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e02b631e1fb4df097767a3ad243f0f7, State: Initialized, Role: FOLLOWER
I20260929 03:15:38.309376 19365 consensus_queue.cc:260] T b722f8ed17134c7aa715550c54a88850 P 3e02b631e1fb4df097767a3ad243f0f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ce84da4de540798b52c01cf4e2161a" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38583 } } peers { permanent_uuid: "3e02b631e1fb4df097767a3ad243f0f7" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 44637 } } peers { permanent_uuid: "aa57c3482e0c4ac3abfcbe2ff72bcacd" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43375 } }
I20260929 03:15:38.309659 19364 raft_consensus.cc:348] T b722f8ed17134c7aa715550c54a88850 P aa57c3482e0c4ac3abfcbe2ff72bcacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ce84da4de540798b52c01cf4e2161a" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38583 } } peers { permanent_uuid: "3e02b631e1fb4df097767a3ad243f0f7" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 44637 } } peers { permanent_uuid: "aa57c3482e0c4ac3abfcbe2ff72bcacd" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43375 } }
I20260929 03:15:38.309926 19364 raft_consensus.cc:374] T b722f8ed17134c7aa715550c54a88850 P aa57c3482e0c4ac3abfcbe2ff72bcacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:38.309960 19085 heartbeater.cc:503] Master 127.13.238.190:43131 was elected leader, sending a full tablet report...
I20260929 03:15:38.309991 19364 raft_consensus.cc:729] T b722f8ed17134c7aa715550c54a88850 P aa57c3482e0c4ac3abfcbe2ff72bcacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa57c3482e0c4ac3abfcbe2ff72bcacd, State: Initialized, Role: FOLLOWER
I20260929 03:15:38.310454 19364 consensus_queue.cc:260] T b722f8ed17134c7aa715550c54a88850 P aa57c3482e0c4ac3abfcbe2ff72bcacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ce84da4de540798b52c01cf4e2161a" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38583 } } peers { permanent_uuid: "3e02b631e1fb4df097767a3ad243f0f7" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 44637 } } peers { permanent_uuid: "aa57c3482e0c4ac3abfcbe2ff72bcacd" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43375 } }
I20260929 03:15:38.311075 19216 heartbeater.cc:503] Master 127.13.238.190:43131 was elected leader, sending a full tablet report...
I20260929 03:15:38.311220 19366 raft_consensus.cc:348] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ce84da4de540798b52c01cf4e2161a" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38583 } } peers { permanent_uuid: "3e02b631e1fb4df097767a3ad243f0f7" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 44637 } } peers { permanent_uuid: "aa57c3482e0c4ac3abfcbe2ff72bcacd" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43375 } }
I20260929 03:15:38.311440 19366 raft_consensus.cc:374] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:38.311491 19366 raft_consensus.cc:729] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2ce84da4de540798b52c01cf4e2161a, State: Initialized, Role: FOLLOWER
I20260929 03:15:38.311615 19365 ts_tablet_manager.cc:1428] T b722f8ed17134c7aa715550c54a88850 P 3e02b631e1fb4df097767a3ad243f0f7: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20260929 03:15:38.311920 19366 consensus_queue.cc:260] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ce84da4de540798b52c01cf4e2161a" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38583 } } peers { permanent_uuid: "3e02b631e1fb4df097767a3ad243f0f7" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 44637 } } peers { permanent_uuid: "aa57c3482e0c4ac3abfcbe2ff72bcacd" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43375 } }
I20260929 03:15:38.312426 19364 ts_tablet_manager.cc:1428] T b722f8ed17134c7aa715550c54a88850 P aa57c3482e0c4ac3abfcbe2ff72bcacd: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 03:15:38.312789 19366 ts_tablet_manager.cc:1428] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 03:15:38.313089 19347 heartbeater.cc:503] Master 127.13.238.190:43131 was elected leader, sending a full tablet report...
W20260929 03:15:38.331537 19217 tablet.cc:2369] T b722f8ed17134c7aa715550c54a88850 P aa57c3482e0c4ac3abfcbe2ff72bcacd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:15:38.393350 19372 raft_consensus.cc:482] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:15:38.393570 19372 raft_consensus.cc:504] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ce84da4de540798b52c01cf4e2161a" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38583 } } peers { permanent_uuid: "3e02b631e1fb4df097767a3ad243f0f7" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 44637 } } peers { permanent_uuid: "aa57c3482e0c4ac3abfcbe2ff72bcacd" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43375 } }
I20260929 03:15:38.394874 19372 leader_election.cc:288] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e02b631e1fb4df097767a3ad243f0f7 (127.13.238.129:44637), aa57c3482e0c4ac3abfcbe2ff72bcacd (127.13.238.130:43375)
I20260929 03:15:38.396827 19349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:38.395134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:43375 (local address 127.13.238.131:46117)
0929 03:15:38.395290 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:38.395307 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:38.395400 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:38.395651 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:38.395654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:38.395671 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:38.396194 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:15:38.396196 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:38.396208 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:15:38.396235 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:38.396550 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:38.396553 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:38.396557 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:15:38.396630 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 03:15:38.396731 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 03:15:38.396802 19363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:38.395265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:46117 (local address 127.13.238.130:43375)
0929 03:15:38.395419 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:38.395424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:38.395443 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:15:38.395494 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:38.395498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:38.395536 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:38.395612 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:38.396401 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:38.396406 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:38.396408 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:15:38.396421 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:38.396465 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:15:38.396477 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:38.396567 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:38.396629 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 03:15:38.396666 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 03:15:38.397043 19362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:38.395332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:48241 (local address 127.13.238.129:44637)
0929 03:15:38.395468 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:38.395472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:38.395846 (+ 374us) server_negotiation.cc:407] Connection header received
0929 03:15:38.395901 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:38.395904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:38.395944 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:38.396026 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:38.396666 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:38.396670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:38.396672 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:15:38.396685 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:38.396702 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:15:38.396709 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:38.396784 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:38.396853 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 03:15:38.396894 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:15:38.397341 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:38.395135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:44637 (local address 127.13.238.131:48241)
0929 03:15:38.395733 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:38.395746 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:38.395843 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:38.396098 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:38.396100 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:38.396116 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:38.396511 (+ 395us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:15:38.396513 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:38.396524 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:15:38.396552 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:38.396780 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:38.396783 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:38.396786 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:15:38.396854 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 03:15:38.397217 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":244,"thread_start_us":133,"threads_started":1}
I20260929 03:15:38.397464 19170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b722f8ed17134c7aa715550c54a88850" candidate_uuid: "e2ce84da4de540798b52c01cf4e2161a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa57c3482e0c4ac3abfcbe2ff72bcacd" is_pre_election: true
I20260929 03:15:38.397819 19170 raft_consensus.cc:2454] T b722f8ed17134c7aa715550c54a88850 P aa57c3482e0c4ac3abfcbe2ff72bcacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2ce84da4de540798b52c01cf4e2161a in term 0.
I20260929 03:15:38.397959 19039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b722f8ed17134c7aa715550c54a88850" candidate_uuid: "e2ce84da4de540798b52c01cf4e2161a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e02b631e1fb4df097767a3ad243f0f7" is_pre_election: true
I20260929 03:15:38.398272 19039 raft_consensus.cc:2454] T b722f8ed17134c7aa715550c54a88850 P 3e02b631e1fb4df097767a3ad243f0f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2ce84da4de540798b52c01cf4e2161a in term 0.
I20260929 03:15:38.398344 19234 leader_election.cc:302] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa57c3482e0c4ac3abfcbe2ff72bcacd, e2ce84da4de540798b52c01cf4e2161a; no voters:
I20260929 03:15:38.398619 19372 raft_consensus.cc:2781] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:15:38.398701 19372 raft_consensus.cc:482] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:15:38.398751 19372 raft_consensus.cc:3037] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:38.399761 19372 raft_consensus.cc:504] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ce84da4de540798b52c01cf4e2161a" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38583 } } peers { permanent_uuid: "3e02b631e1fb4df097767a3ad243f0f7" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 44637 } } peers { permanent_uuid: "aa57c3482e0c4ac3abfcbe2ff72bcacd" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43375 } }
I20260929 03:15:38.400097 19372 leader_election.cc:288] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a [CANDIDATE]: Term 1 election: Requested vote from peers 3e02b631e1fb4df097767a3ad243f0f7 (127.13.238.129:44637), aa57c3482e0c4ac3abfcbe2ff72bcacd (127.13.238.130:43375)
I20260929 03:15:38.400449 19170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b722f8ed17134c7aa715550c54a88850" candidate_uuid: "e2ce84da4de540798b52c01cf4e2161a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa57c3482e0c4ac3abfcbe2ff72bcacd"
I20260929 03:15:38.400537 19039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b722f8ed17134c7aa715550c54a88850" candidate_uuid: "e2ce84da4de540798b52c01cf4e2161a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e02b631e1fb4df097767a3ad243f0f7"
I20260929 03:15:38.400581 19170 raft_consensus.cc:3037] T b722f8ed17134c7aa715550c54a88850 P aa57c3482e0c4ac3abfcbe2ff72bcacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:38.400650 19039 raft_consensus.cc:3037] T b722f8ed17134c7aa715550c54a88850 P 3e02b631e1fb4df097767a3ad243f0f7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:38.401850 19039 raft_consensus.cc:2454] T b722f8ed17134c7aa715550c54a88850 P 3e02b631e1fb4df097767a3ad243f0f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2ce84da4de540798b52c01cf4e2161a in term 1.
I20260929 03:15:38.401850 19170 raft_consensus.cc:2454] T b722f8ed17134c7aa715550c54a88850 P aa57c3482e0c4ac3abfcbe2ff72bcacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2ce84da4de540798b52c01cf4e2161a in term 1.
I20260929 03:15:38.402105 19234 leader_election.cc:302] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa57c3482e0c4ac3abfcbe2ff72bcacd, e2ce84da4de540798b52c01cf4e2161a; no voters:
I20260929 03:15:38.402292 19372 raft_consensus.cc:2781] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:38.402498 19372 raft_consensus.cc:686] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a [term 1 LEADER]: Becoming Leader. State: Replica: e2ce84da4de540798b52c01cf4e2161a, State: Running, Role: LEADER
I20260929 03:15:38.402832 19372 consensus_queue.cc:237] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ce84da4de540798b52c01cf4e2161a" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38583 } } peers { permanent_uuid: "3e02b631e1fb4df097767a3ad243f0f7" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 44637 } } peers { permanent_uuid: "aa57c3482e0c4ac3abfcbe2ff72bcacd" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43375 } }
I20260929 03:15:38.405942 18895 catalog_manager.cc:5495] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a reported cstate change: term changed from 0 to 1, leader changed from <none> to e2ce84da4de540798b52c01cf4e2161a (127.13.238.131). New cstate: current_term: 1 leader_uuid: "e2ce84da4de540798b52c01cf4e2161a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ce84da4de540798b52c01cf4e2161a" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e02b631e1fb4df097767a3ad243f0f7" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 44637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa57c3482e0c4ac3abfcbe2ff72bcacd" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43375 } health_report { overall_health: UNKNOWN } } }
W20260929 03:15:38.408862 19086 tablet.cc:2369] T b722f8ed17134c7aa715550c54a88850 P 3e02b631e1fb4df097767a3ad243f0f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:15:38.441388 19360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:38.439962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38502 (local address 127.13.238.131:38583)
0929 03:15:38.440095 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:38.440099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:38.440114 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:15:38.440218 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:38.440221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:38.440291 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:38.440360 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:38.441032 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:38.441036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:38.441039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:15:38.441051 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:38.441070 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:15:38.441077 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:38.441150 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:38.441219 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 03:15:38.441259 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:15:38.447268 19170 raft_consensus.cc:1261] T b722f8ed17134c7aa715550c54a88850 P aa57c3482e0c4ac3abfcbe2ff72bcacd [term 1 FOLLOWER]: Refusing update from remote peer e2ce84da4de540798b52c01cf4e2161a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:38.447268 19039 raft_consensus.cc:1261] T b722f8ed17134c7aa715550c54a88850 P 3e02b631e1fb4df097767a3ad243f0f7 [term 1 FOLLOWER]: Refusing update from remote peer e2ce84da4de540798b52c01cf4e2161a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:38.447942 19374 consensus_queue.cc:1035] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa57c3482e0c4ac3abfcbe2ff72bcacd" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:38.448117 19372 consensus_queue.cc:1035] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e02b631e1fb4df097767a3ad243f0f7" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 44637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:38.455793 19379 mvcc.cc:205] Tried to move back new op lower bound from 7334509520672960512 to 7334509520500629504. Current Snapshot: MvccSnapshot[applied={T|T < 7334509520672960512}]
I20260929 03:15:38.456339 19378 mvcc.cc:205] Tried to move back new op lower bound from 7334509520672960512 to 7334509520500629504. Current Snapshot: MvccSnapshot[applied={T|T < 7334509520672960512}]
I20260929 03:15:38.463601 18894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:38.463747 18894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:38.466004 18894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1013eee3bbd94d93956a627be5ac5f06: Sending DeleteTablet for 3 replicas of tablet b722f8ed17134c7aa715550c54a88850
I20260929 03:15:38.466747 19281 tablet_service.cc:1514] Processing DeleteTablet for tablet b722f8ed17134c7aa715550c54a88850 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:38 UTC) from {username='slave'} at 127.0.0.1:38496
I20260929 03:15:38.466831 19150 tablet_service.cc:1514] Processing DeleteTablet for tablet b722f8ed17134c7aa715550c54a88850 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:38 UTC) from {username='slave'} at 127.0.0.1:33920
I20260929 03:15:38.466879 19019 tablet_service.cc:1514] Processing DeleteTablet for tablet b722f8ed17134c7aa715550c54a88850 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:38 UTC) from {username='slave'} at 127.0.0.1:44438
I20260929 03:15:38.467417 19386 tablet_replica.cc:330] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a: stopping tablet replica
I20260929 03:15:38.467749 19386 raft_consensus.cc:2229] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:15:38.468015 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 18957
I20260929 03:15:38.468752 19386 raft_consensus.cc:2258] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:38.470621 19386 ts_tablet_manager.cc:1905] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:38.473346 19386 ts_tablet_manager.cc:1918] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:15:38.473515 19386 log.cc:1196] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/b722f8ed17134c7aa715550c54a88850
I20260929 03:15:38.473565 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 19088
W20260929 03:15:38.473541 18883 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.238.129:44637 (error 108)
I20260929 03:15:38.474879 19359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:38.474583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:44637 (local address 127.0.0.1:44454)
0929 03:15:38.474712 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:38.474787 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.129:44637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 03:15:38.475042 18883 catalog_manager.cc:4584] TS 3e02b631e1fb4df097767a3ad243f0f7 (127.13.238.129:44637): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b722f8ed17134c7aa715550c54a88850: Network error: Client connection negotiation failed: client connection to 127.13.238.129:44637: connect: Connection refused (error 111)
I20260929 03:15:38.478874 19386 ts_tablet_manager.cc:1939] T b722f8ed17134c7aa715550c54a88850 P e2ce84da4de540798b52c01cf4e2161a: Deleting consensus metadata
I20260929 03:15:38.478986 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 19219
I20260929 03:15:38.480549 19359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:38.480165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:43375 (local address 127.0.0.1:33930)
0929 03:15:38.480406 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:38.480463 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.130:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260929 03:15:38.484174 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 18866
2026-09-29T03:15:38Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1002 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 03:15:38.504197 14266 mini_oidc.cc:152] Starting JWKS server
I20260929 03:15:38.505815 14266 webserver.cc:460] Webserver started at https://127.0.0.1:42551/ using document root <none> and password file <none>
I20260929 03:15:38.505959 14266 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 03:15:38.506179 14266 webserver.cc:460] Webserver started at http://127.0.0.1:40359/ using document root <none> and password file <none>
2026-09-29T03:15:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:15:38Z Disabled control of system clock
I20260929 03:15:38.514511 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:41989
--webserver_interface=127.13.238.190
--webserver_port=0
--builtin_ntp_servers=127.13.238.148:41651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:41989
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42551/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651713532531-14266-0/testchainca.cert with env {}
W20260929 03:15:38.622076 19395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:38.622350 19395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:38.622395 19395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:38.626403 19395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:38.626454 19395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:38.626467 19395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:38.626482 19395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:38.626500 19395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 03:15:38.626530 19395 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 03:15:38.626554 19395 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42551/jwks.json
I20260929 03:15:38.631779 19395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:41651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:41989
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:41989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42551/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651713532531-14266-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:38.632817 19395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:38.633832 19395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:38.642139 19401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:38.642170 19403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:38.642313 19395 server_base.cc:1034] running on GCE node
W20260929 03:15:38.642182 19400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:38.642959 19395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:38.644074 19395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:38.645241 19395 hybrid_clock.cc:648] HybridClock initialized: now 1790651738645221 us; error 29 us; skew 500 ppm
I20260929 03:15:38.647126 19395 webserver.cc:460] Webserver started at http://127.13.238.190:34043/ using document root <none> and password file <none>
I20260929 03:15:38.647689 19395 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:38.647760 19395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:38.647953 19395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:38.650063 19395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "020bd8cb149743388c1afa2677e91540"
format_stamp: "Formatted at 2026-09-29 03:15:38 on dist-test-slave-nbjg"
server_key: "b962fdc26564e229b67fcecec64e1664"
server_key_iv: "fa83521f8781af37f18e5bbe2e82e389"
server_key_version: "encryptionkey@0"
I20260929 03:15:38.650633 19395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "020bd8cb149743388c1afa2677e91540"
format_stamp: "Formatted at 2026-09-29 03:15:38 on dist-test-slave-nbjg"
server_key: "b962fdc26564e229b67fcecec64e1664"
server_key_iv: "fa83521f8781af37f18e5bbe2e82e389"
server_key_version: "encryptionkey@0"
I20260929 03:15:38.654182 19395 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:15:38.656574 19409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:38.657776 19395 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 03:15:38.657881 19395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "020bd8cb149743388c1afa2677e91540"
format_stamp: "Formatted at 2026-09-29 03:15:38 on dist-test-slave-nbjg"
server_key: "b962fdc26564e229b67fcecec64e1664"
server_key_iv: "fa83521f8781af37f18e5bbe2e82e389"
server_key_version: "encryptionkey@0"
I20260929 03:15:38.657979 19395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:38.673942 19395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:38.674758 19395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:38.674949 19395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:38.681625 19395 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:41989
I20260929 03:15:38.681648 19461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:41989 every 8 connection(s)
I20260929 03:15:38.682578 19395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:15:38.685719 19462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:38.689895 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 19395
I20260929 03:15:38.690021 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:15:38.690325 14266 external_mini_cluster.cc:1428] Setting key 9348d7e84f4ec8039c55e4e4ec643c4e
I20260929 03:15:38.691639 19462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 020bd8cb149743388c1afa2677e91540: Bootstrap starting.
I20260929 03:15:38.693918 19462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 020bd8cb149743388c1afa2677e91540: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:38.694684 19462 log.cc:824] T 00000000000000000000000000000000 P 020bd8cb149743388c1afa2677e91540: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:38.696498 19462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 020bd8cb149743388c1afa2677e91540: No bootstrap required, opened a new log
I20260929 03:15:38.699558 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:38.691736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52496 (local address 127.13.238.190:41989)
0929 03:15:38.692427 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:38.692442 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:38.692486 (+ 44us) server_negotiation.cc:407] Connection header received
0929 03:15:38.693110 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:38.693126 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:38.693525 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:38.693847 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:38.694325 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:38.695540 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:38.696419 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:38.696926 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:38.697072 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:38.697850 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:38.697855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:38.697869 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:15:38.698290 (+ 421us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:38.698346 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:15:38.698367 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:38.698451 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:38.698572 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 03:15:38.698834 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":142,"threads_started":1}
I20260929 03:15:38.700068 19462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 020bd8cb149743388c1afa2677e91540 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020bd8cb149743388c1afa2677e91540" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 41989 } }
I20260929 03:15:38.700348 19462 raft_consensus.cc:374] T 00000000000000000000000000000000 P 020bd8cb149743388c1afa2677e91540 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:38.700402 19462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 020bd8cb149743388c1afa2677e91540 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 020bd8cb149743388c1afa2677e91540, State: Initialized, Role: FOLLOWER
I20260929 03:15:38.700850 19462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 020bd8cb149743388c1afa2677e91540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020bd8cb149743388c1afa2677e91540" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 41989 } }
I20260929 03:15:38.700976 19462 raft_consensus.cc:388] T 00000000000000000000000000000000 P 020bd8cb149743388c1afa2677e91540 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:15:38.701036 19462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 020bd8cb149743388c1afa2677e91540 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:15:38.701133 19462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 020bd8cb149743388c1afa2677e91540 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:38.702171 19462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 020bd8cb149743388c1afa2677e91540 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020bd8cb149743388c1afa2677e91540" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 41989 } }
I20260929 03:15:38.702556 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:15:38.702570 19462 leader_election.cc:302] T 00000000000000000000000000000000 P 020bd8cb149743388c1afa2677e91540 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 020bd8cb149743388c1afa2677e91540; no voters:
I20260929 03:15:38.702885 19462 leader_election.cc:288] T 00000000000000000000000000000000 P 020bd8cb149743388c1afa2677e91540 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:15:38.702986 19467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 020bd8cb149743388c1afa2677e91540 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:38.703145 19467 raft_consensus.cc:686] T 00000000000000000000000000000000 P 020bd8cb149743388c1afa2677e91540 [term 1 LEADER]: Becoming Leader. State: Replica: 020bd8cb149743388c1afa2677e91540, State: Running, Role: LEADER
I20260929 03:15:38.703455 19467 consensus_queue.cc:237] T 00000000000000000000000000000000 P 020bd8cb149743388c1afa2677e91540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020bd8cb149743388c1afa2677e91540" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 41989 } }
I20260929 03:15:38.704208 19462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 020bd8cb149743388c1afa2677e91540 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:38.704751 19467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 020bd8cb149743388c1afa2677e91540 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 020bd8cb149743388c1afa2677e91540. Latest consensus state: current_term: 1 leader_uuid: "020bd8cb149743388c1afa2677e91540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020bd8cb149743388c1afa2677e91540" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 41989 } } }
I20260929 03:15:38.704900 19467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 020bd8cb149743388c1afa2677e91540 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:38.705499 19468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 020bd8cb149743388c1afa2677e91540 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "020bd8cb149743388c1afa2677e91540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020bd8cb149743388c1afa2677e91540" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 41989 } } }
I20260929 03:15:38.705600 19468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 020bd8cb149743388c1afa2677e91540 [sys.catalog]: This master's current role is: LEADER
W20260929 03:15:38.708849 19482 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 020bd8cb149743388c1afa2677e91540: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:15:38.708923 19482 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:15:38.708977 19472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:15:38.709661 19472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:15:38.715185 19472 catalog_manager.cc:1347] Generated new cluster ID: fce080e221a64c7fb3f1b45792734b86
I20260929 03:15:38.715252 19472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:15:38.728161 19472 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:15:38.729032 19472 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:15:38.740661 19472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 020bd8cb149743388c1afa2677e91540: Generated new TSK 0
I20260929 03:15:38.741346 19472 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 03:15:38.772826 19492 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 03:15:38.773000 19492 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 03:15:38.773641 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:38.755619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52512 (local address 127.13.238.190:41989)
0929 03:15:38.755746 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:38.755751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:38.755924 (+ 173us) server_negotiation.cc:407] Connection header received
0929 03:15:38.756123 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:38.756127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:38.756187 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:38.756301 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0929 03:15:38.756497 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:38.757330 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:38.758167 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:38.758557 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:38.758659 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:38.758946 (+ 287us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 03:15:38.773168 (+ 14222us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42551/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 03:15:38.773489 (+ 321us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52512: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42551/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 03:15:38.773736 19465 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52512: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42551/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 03:15:38.774991 19491 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:38.755523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:41989 (local address 127.0.0.1:52512)
0929 03:15:38.755851 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:38.755863 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:38.755983 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:38.756307 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:38.756310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:38.756320 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0929 03:15:38.756410 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:38.756415 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:38.757432 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:38.757434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:38.758068 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:38.758076 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:38.758657 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:38.758659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:38.758721 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:38.758766 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 03:15:38.773455 (+ 14689us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42551/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 03:15:38.773809 (+ 354us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.238.190:41989: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42551/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":103,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.13.238.190:41989: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42551/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 03:15:38.775946 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 19395
2026-09-29T03:15:38Z chronyd exiting
I20260929 03:15:38.812748 14266 test_util.cc:175] -----------------------------------------------
I20260929 03:15:38.812837 14266 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790651713532531-14266-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (310 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 03:15:38.813831 14266 mini_oidc.cc:152] Starting JWKS server
I20260929 03:15:38.814738 14266 webserver.cc:460] Webserver started at https://127.0.0.1:36571/ using document root <none> and password file <none>
I20260929 03:15:38.814837 14266 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 03:15:38.815091 14266 webserver.cc:460] Webserver started at http://127.0.0.1:39305/ using document root <none> and password file <none>
2026-09-29T03:15:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:15:38Z Disabled control of system clock
I20260929 03:15:38.823364 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:34877
--webserver_interface=127.13.238.190
--webserver_port=0
--builtin_ntp_servers=127.13.238.148:37711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:34877
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36571/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651713532531-14266-0/testchainca.cert with env {}
W20260929 03:15:38.931277 19498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:38.931543 19498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:38.931581 19498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:38.935547 19498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:38.935602 19498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:38.935617 19498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:38.935633 19498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:38.935647 19498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 03:15:38.935664 19498 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 03:15:38.935683 19498 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36571/jwks.json
I20260929 03:15:38.940972 19498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:37711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:34877
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:34877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36571/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651713532531-14266-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:38.942075 19498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:38.943157 19498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:38.951647 19504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:38.951766 19506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:38.951700 19498 server_base.cc:1034] running on GCE node
W20260929 03:15:38.951647 19503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:38.952571 19498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:38.953711 19498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:38.954866 19498 hybrid_clock.cc:648] HybridClock initialized: now 1790651738954846 us; error 31 us; skew 500 ppm
I20260929 03:15:38.956779 19498 webserver.cc:460] Webserver started at http://127.13.238.190:46093/ using document root <none> and password file <none>
I20260929 03:15:38.957358 19498 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:38.957440 19498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:38.957680 19498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:38.959796 19498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "fd1e2144e4a444d0a331c89da5e68872"
format_stamp: "Formatted at 2026-09-29 03:15:38 on dist-test-slave-nbjg"
server_key: "cd12d510b72dde4c00a54a1744d48594"
server_key_iv: "e89fb7f3477c8f55eb9dbdfa4b156d43"
server_key_version: "encryptionkey@0"
I20260929 03:15:38.960350 19498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "fd1e2144e4a444d0a331c89da5e68872"
format_stamp: "Formatted at 2026-09-29 03:15:38 on dist-test-slave-nbjg"
server_key: "cd12d510b72dde4c00a54a1744d48594"
server_key_iv: "e89fb7f3477c8f55eb9dbdfa4b156d43"
server_key_version: "encryptionkey@0"
I20260929 03:15:38.964036 19498 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 03:15:38.966439 19512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:38.967697 19498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 03:15:38.967813 19498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "fd1e2144e4a444d0a331c89da5e68872"
format_stamp: "Formatted at 2026-09-29 03:15:38 on dist-test-slave-nbjg"
server_key: "cd12d510b72dde4c00a54a1744d48594"
server_key_iv: "e89fb7f3477c8f55eb9dbdfa4b156d43"
server_key_version: "encryptionkey@0"
I20260929 03:15:38.967909 19498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:38.998158 19498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:38.998965 19498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:38.999146 19498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:39.005874 19498 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:34877
I20260929 03:15:39.005903 19564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:34877 every 8 connection(s)
I20260929 03:15:39.006897 19498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:15:39.008654 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 19498
I20260929 03:15:39.008796 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:15:39.009119 14266 external_mini_cluster.cc:1428] Setting key e738ff3a9d07f4662a8f603d6efeafbe
I20260929 03:15:39.010924 19565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:39.016947 19565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd1e2144e4a444d0a331c89da5e68872: Bootstrap starting.
I20260929 03:15:39.018703 19568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:39.010561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45450 (local address 127.13.238.190:34877)
0929 03:15:39.011325 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:39.011343 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:39.011381 (+ 38us) server_negotiation.cc:407] Connection header received
0929 03:15:39.012079 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:39.012098 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:39.012523 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:39.012853 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:39.013311 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:39.014367 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:39.015312 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:39.015829 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:39.015983 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:39.016852 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:39.016857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:39.016871 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:15:39.017291 (+ 420us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:39.017350 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:15:39.017372 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:39.017472 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:39.017615 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 03:15:39.017894 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":156,"threads_started":1}
I20260929 03:15:39.019379 19565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd1e2144e4a444d0a331c89da5e68872: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:39.020102 19565 log.cc:824] T 00000000000000000000000000000000 P fd1e2144e4a444d0a331c89da5e68872: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:39.021636 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:15:39.022396 19565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd1e2144e4a444d0a331c89da5e68872: No bootstrap required, opened a new log
I20260929 03:15:39.026341 19565 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd1e2144e4a444d0a331c89da5e68872 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1e2144e4a444d0a331c89da5e68872" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 34877 } }
I20260929 03:15:39.026628 19565 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd1e2144e4a444d0a331c89da5e68872 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:39.026692 19565 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd1e2144e4a444d0a331c89da5e68872 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd1e2144e4a444d0a331c89da5e68872, State: Initialized, Role: FOLLOWER
I20260929 03:15:39.027107 19565 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd1e2144e4a444d0a331c89da5e68872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1e2144e4a444d0a331c89da5e68872" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 34877 } }
I20260929 03:15:39.027233 19565 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd1e2144e4a444d0a331c89da5e68872 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:15:39.027297 19565 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd1e2144e4a444d0a331c89da5e68872 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:15:39.027391 19565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd1e2144e4a444d0a331c89da5e68872 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:39.028316 19565 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd1e2144e4a444d0a331c89da5e68872 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1e2144e4a444d0a331c89da5e68872" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 34877 } }
I20260929 03:15:39.028632 19565 leader_election.cc:302] T 00000000000000000000000000000000 P fd1e2144e4a444d0a331c89da5e68872 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd1e2144e4a444d0a331c89da5e68872; no voters:
I20260929 03:15:39.028936 19565 leader_election.cc:288] T 00000000000000000000000000000000 P fd1e2144e4a444d0a331c89da5e68872 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:15:39.029088 19570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd1e2144e4a444d0a331c89da5e68872 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:39.029307 19570 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd1e2144e4a444d0a331c89da5e68872 [term 1 LEADER]: Becoming Leader. State: Replica: fd1e2144e4a444d0a331c89da5e68872, State: Running, Role: LEADER
I20260929 03:15:39.029635 19570 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd1e2144e4a444d0a331c89da5e68872 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1e2144e4a444d0a331c89da5e68872" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 34877 } }
I20260929 03:15:39.030119 19565 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd1e2144e4a444d0a331c89da5e68872 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:39.031252 19572 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd1e2144e4a444d0a331c89da5e68872 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd1e2144e4a444d0a331c89da5e68872. Latest consensus state: current_term: 1 leader_uuid: "fd1e2144e4a444d0a331c89da5e68872" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1e2144e4a444d0a331c89da5e68872" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 34877 } } }
I20260929 03:15:39.031383 19572 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd1e2144e4a444d0a331c89da5e68872 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:39.031683 19571 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd1e2144e4a444d0a331c89da5e68872 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd1e2144e4a444d0a331c89da5e68872" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1e2144e4a444d0a331c89da5e68872" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 34877 } } }
I20260929 03:15:39.031781 19571 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd1e2144e4a444d0a331c89da5e68872 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:39.032104 19579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:15:39.035128 19579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:15:39.040580 19579 catalog_manager.cc:1347] Generated new cluster ID: dd8b8b5b864d4dd380b28c63051b2c50
I20260929 03:15:39.040640 19579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:15:39.047715 19579 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:15:39.048838 19579 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:15:39.054436 19579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd1e2144e4a444d0a331c89da5e68872: Generated new TSK 0
I20260929 03:15:39.055174 19579 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 03:15:39.087277 19595 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 03:15:39.087360 19595 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 03:15:39.088183 19568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:39.068168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45464 (local address 127.13.238.190:34877)
0929 03:15:39.068292 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:39.068296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:39.068476 (+ 180us) server_negotiation.cc:407] Connection header received
0929 03:15:39.068641 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:39.068644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:39.068690 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:39.068769 (+ 79us) server_negotiation.cc:226] Negotiated authn=JWT
0929 03:15:39.068995 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:39.069525 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:39.070740 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:39.070956 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:39.071035 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:39.071342 (+ 307us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 03:15:39.087715 (+ 16373us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36571/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 03:15:39.088027 (+ 312us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45464: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36571/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 03:15:39.088256 19568 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45464: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36571/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 03:15:39.088254 19594 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:39.068096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:34877 (local address 127.0.0.1:45464)
0929 03:15:39.068393 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:39.068408 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:39.068520 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:39.068787 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:39.068791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:39.068799 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 03:15:39.068909 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:39.068916 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:39.069616 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:39.069620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:39.070640 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:39.070650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:39.071066 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:39.071069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:39.071162 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:39.071190 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 03:15:39.087959 (+ 16769us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36571/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 03:15:39.088109 (+ 150us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.238.190:34877: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36571/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":88,"threads_started":1}
I20260929 03:15:39.088687 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 19498
2026-09-29T03:15:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (313 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 03:15:39.127641 14266 mini_oidc.cc:152] Starting JWKS server
I20260929 03:15:39.128471 14266 webserver.cc:460] Webserver started at https://127.0.0.1:41877/ using document root <none> and password file <none>
I20260929 03:15:39.128576 14266 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 03:15:39.128839 14266 webserver.cc:460] Webserver started at http://127.0.0.1:32987/ using document root <none> and password file <none>
2026-09-29T03:15:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:15:39Z Disabled control of system clock
I20260929 03:15:39.137210 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:41501
--webserver_interface=127.13.238.190
--webserver_port=0
--builtin_ntp_servers=127.13.238.148:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:41501
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41877/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 03:15:39.248639 19601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:39.248909 19601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:39.248944 19601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:39.252990 19601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:39.253048 19601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:39.253062 19601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:39.253079 19601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:39.253098 19601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 03:15:39.253118 19601 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 03:15:39.253134 19601 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41877/jwks.json
I20260929 03:15:39.257867 19601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:41501
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:41501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41877/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:39.258953 19601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:39.260329 19601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:39.268548 19607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:39.268627 19601 server_base.cc:1034] running on GCE node
W20260929 03:15:39.268553 19606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:39.268558 19609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:39.269337 19601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:39.270424 19601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:39.271574 19601 hybrid_clock.cc:648] HybridClock initialized: now 1790651739271556 us; error 30 us; skew 500 ppm
I20260929 03:15:39.273345 19601 webserver.cc:460] Webserver started at http://127.13.238.190:46571/ using document root <none> and password file <none>
I20260929 03:15:39.273875 19601 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:39.273933 19601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:39.274114 19601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:39.276152 19601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "af1fcca53c4e44ab98b8106c6a28e894"
format_stamp: "Formatted at 2026-09-29 03:15:39 on dist-test-slave-nbjg"
server_key: "941a9e1406871f392a11da7a842f9750"
server_key_iv: "9257211f16b98dab49db08e2a646e044"
server_key_version: "encryptionkey@0"
I20260929 03:15:39.276685 19601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "af1fcca53c4e44ab98b8106c6a28e894"
format_stamp: "Formatted at 2026-09-29 03:15:39 on dist-test-slave-nbjg"
server_key: "941a9e1406871f392a11da7a842f9750"
server_key_iv: "9257211f16b98dab49db08e2a646e044"
server_key_version: "encryptionkey@0"
I20260929 03:15:39.280372 19601 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 03:15:39.282691 19615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:39.283867 19601 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:39.283972 19601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "af1fcca53c4e44ab98b8106c6a28e894"
format_stamp: "Formatted at 2026-09-29 03:15:39 on dist-test-slave-nbjg"
server_key: "941a9e1406871f392a11da7a842f9750"
server_key_iv: "9257211f16b98dab49db08e2a646e044"
server_key_version: "encryptionkey@0"
I20260929 03:15:39.284072 19601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:39.303027 19601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:39.303715 19601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:39.303885 19601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:39.310436 19601 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:41501
I20260929 03:15:39.310460 19667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:41501 every 8 connection(s)
I20260929 03:15:39.311365 19601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:15:39.311961 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 19601
I20260929 03:15:39.312077 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:15:39.312338 14266 external_mini_cluster.cc:1428] Setting key be30b43e2cad3513003bf050ae05bd7a
I20260929 03:15:39.315004 19668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:39.321907 19668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af1fcca53c4e44ab98b8106c6a28e894: Bootstrap starting.
I20260929 03:15:39.322758 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:39.313715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52580 (local address 127.13.238.190:41501)
0929 03:15:39.314454 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:39.314473 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:39.314519 (+ 46us) server_negotiation.cc:407] Connection header received
0929 03:15:39.315384 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:39.315410 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:39.315921 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:39.316290 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:39.316749 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:39.318241 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:39.319120 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:39.319792 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:39.319958 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:39.320715 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:39.320722 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:39.320738 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:15:39.321240 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:39.321311 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:15:39.321349 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:39.321440 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:39.321534 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 03:15:39.321801 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":132,"threads_started":1}
I20260929 03:15:39.324519 19668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af1fcca53c4e44ab98b8106c6a28e894: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:39.325321 19668 log.cc:824] T 00000000000000000000000000000000 P af1fcca53c4e44ab98b8106c6a28e894: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:39.325757 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:15:39.327476 19668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af1fcca53c4e44ab98b8106c6a28e894: No bootstrap required, opened a new log
I20260929 03:15:39.330868 19668 raft_consensus.cc:348] T 00000000000000000000000000000000 P af1fcca53c4e44ab98b8106c6a28e894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1fcca53c4e44ab98b8106c6a28e894" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 41501 } }
I20260929 03:15:39.331084 19668 raft_consensus.cc:374] T 00000000000000000000000000000000 P af1fcca53c4e44ab98b8106c6a28e894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:39.331146 19668 raft_consensus.cc:729] T 00000000000000000000000000000000 P af1fcca53c4e44ab98b8106c6a28e894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af1fcca53c4e44ab98b8106c6a28e894, State: Initialized, Role: FOLLOWER
I20260929 03:15:39.331532 19668 consensus_queue.cc:260] T 00000000000000000000000000000000 P af1fcca53c4e44ab98b8106c6a28e894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1fcca53c4e44ab98b8106c6a28e894" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 41501 } }
I20260929 03:15:39.331652 19668 raft_consensus.cc:388] T 00000000000000000000000000000000 P af1fcca53c4e44ab98b8106c6a28e894 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:15:39.331713 19668 raft_consensus.cc:482] T 00000000000000000000000000000000 P af1fcca53c4e44ab98b8106c6a28e894 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:15:39.331808 19668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af1fcca53c4e44ab98b8106c6a28e894 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:39.332643 19668 raft_consensus.cc:504] T 00000000000000000000000000000000 P af1fcca53c4e44ab98b8106c6a28e894 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1fcca53c4e44ab98b8106c6a28e894" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 41501 } }
I20260929 03:15:39.332958 19668 leader_election.cc:302] T 00000000000000000000000000000000 P af1fcca53c4e44ab98b8106c6a28e894 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af1fcca53c4e44ab98b8106c6a28e894; no voters:
I20260929 03:15:39.333256 19668 leader_election.cc:288] T 00000000000000000000000000000000 P af1fcca53c4e44ab98b8106c6a28e894 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:15:39.333348 19673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af1fcca53c4e44ab98b8106c6a28e894 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:39.333559 19673 raft_consensus.cc:686] T 00000000000000000000000000000000 P af1fcca53c4e44ab98b8106c6a28e894 [term 1 LEADER]: Becoming Leader. State: Replica: af1fcca53c4e44ab98b8106c6a28e894, State: Running, Role: LEADER
I20260929 03:15:39.333873 19673 consensus_queue.cc:237] T 00000000000000000000000000000000 P af1fcca53c4e44ab98b8106c6a28e894 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1fcca53c4e44ab98b8106c6a28e894" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 41501 } }
I20260929 03:15:39.334481 19668 sys_catalog.cc:564] T 00000000000000000000000000000000 P af1fcca53c4e44ab98b8106c6a28e894 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:39.335618 19674 sys_catalog.cc:455] T 00000000000000000000000000000000 P af1fcca53c4e44ab98b8106c6a28e894 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af1fcca53c4e44ab98b8106c6a28e894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1fcca53c4e44ab98b8106c6a28e894" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 41501 } } }
I20260929 03:15:39.335659 19675 sys_catalog.cc:455] T 00000000000000000000000000000000 P af1fcca53c4e44ab98b8106c6a28e894 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af1fcca53c4e44ab98b8106c6a28e894. Latest consensus state: current_term: 1 leader_uuid: "af1fcca53c4e44ab98b8106c6a28e894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1fcca53c4e44ab98b8106c6a28e894" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 41501 } } }
I20260929 03:15:39.335820 19674 sys_catalog.cc:458] T 00000000000000000000000000000000 P af1fcca53c4e44ab98b8106c6a28e894 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:39.335850 19675 sys_catalog.cc:458] T 00000000000000000000000000000000 P af1fcca53c4e44ab98b8106c6a28e894 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:39.336349 19684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:15:39.339210 19684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:15:39.344386 19684 catalog_manager.cc:1347] Generated new cluster ID: c6291dd00dd542099fe6fdc05a783a60
I20260929 03:15:39.344452 19684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:15:39.356853 19684 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:15:39.358034 19684 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:15:39.363150 19684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af1fcca53c4e44ab98b8106c6a28e894: Generated new TSK 0
I20260929 03:15:39.363874 19684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:15:39.378548 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:39.373435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52586 (local address 127.13.238.190:41501)
0929 03:15:39.373567 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:39.373572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:39.373694 (+ 122us) server_negotiation.cc:407] Connection header received
0929 03:15:39.373851 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:39.373855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:39.373912 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:39.374020 (+ 108us) server_negotiation.cc:226] Negotiated authn=JWT
0929 03:15:39.374232 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:39.375060 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:39.375940 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:39.376272 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:39.376365 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:39.376650 (+ 285us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 03:15:39.378140 (+ 1490us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41877/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 03:15:39.378400 (+ 260us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52586: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41877/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 03:15:39.378620 19671 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52586: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41877/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 03:15:39.380084 19697 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:39.373338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:41501 (local address 127.0.0.1:52586)
0929 03:15:39.373609 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:39.373621 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:39.373717 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:39.374029 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:39.374031 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:39.374038 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0929 03:15:39.374134 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:39.374139 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:39.375177 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:39.375180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:39.375838 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:39.375846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:39.376391 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:39.376395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:39.376462 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:39.376486 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 03:15:39.378323 (+ 1837us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41877/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 03:15:39.379916 (+ 1593us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.238.190:41501: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41877/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":1419,"thread_start_us":90,"threads_started":1}
I20260929 03:15:39.380656 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 19601
E20260929 03:15:39.391166 19698 webserver.cc:533] Webserver: sslize failed:
E20260929 03:15:39.391237 19698 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T03:15:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (292 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T03:15:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:15:39Z Disabled control of system clock
I20260929 03:15:39.428669 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:39761
--webserver_interface=127.13.238.190
--webserver_port=0
--builtin_ntp_servers=127.13.238.148:44545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:39761
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651713532531-14266-0/testchainca.cert with env {}
W20260929 03:15:39.539091 19702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:39.539361 19702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:39.539402 19702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:39.543429 19702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:39.543485 19702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:39.543499 19702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:39.543514 19702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:39.543529 19702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 03:15:39.543558 19702 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 03:15:39.543583 19702 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 03:15:39.548862 19702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:44545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:39761
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:39761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651713532531-14266-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:39.549894 19702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:39.550938 19702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:15:39.559548 19702 server_base.cc:1034] running on GCE node
W20260929 03:15:39.559494 19707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:39.559494 19708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:39.559516 19710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:39.560381 19702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:39.561565 19702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:39.562760 19702 hybrid_clock.cc:648] HybridClock initialized: now 1790651739562738 us; error 31 us; skew 500 ppm
I20260929 03:15:39.564611 19702 webserver.cc:460] Webserver started at http://127.13.238.190:43905/ using document root <none> and password file <none>
I20260929 03:15:39.565160 19702 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:39.565224 19702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:39.565431 19702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:39.567476 19702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "6ab477838f7c4fa793a171b5a12aed9f"
format_stamp: "Formatted at 2026-09-29 03:15:39 on dist-test-slave-nbjg"
server_key: "d26c34a5dc29c7d7396c1a1db32a494b"
server_key_iv: "e651e5ed241faa1e88be88e54e443a86"
server_key_version: "encryptionkey@0"
I20260929 03:15:39.568025 19702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "6ab477838f7c4fa793a171b5a12aed9f"
format_stamp: "Formatted at 2026-09-29 03:15:39 on dist-test-slave-nbjg"
server_key: "d26c34a5dc29c7d7396c1a1db32a494b"
server_key_iv: "e651e5ed241faa1e88be88e54e443a86"
server_key_version: "encryptionkey@0"
I20260929 03:15:39.571619 19702 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:15:39.573958 19716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:39.575201 19702 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:39.575330 19702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "6ab477838f7c4fa793a171b5a12aed9f"
format_stamp: "Formatted at 2026-09-29 03:15:39 on dist-test-slave-nbjg"
server_key: "d26c34a5dc29c7d7396c1a1db32a494b"
server_key_iv: "e651e5ed241faa1e88be88e54e443a86"
server_key_version: "encryptionkey@0"
I20260929 03:15:39.575428 19702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:39.602309 19702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:39.603035 19702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:39.603209 19702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:39.610131 19702 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:39761
I20260929 03:15:39.610149 19768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:39761 every 8 connection(s)
I20260929 03:15:39.611136 19702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:15:39.614001 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 19702
I20260929 03:15:39.614095 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:15:39.614367 19769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:39.614377 14266 external_mini_cluster.cc:1428] Setting key f8461e8ff603edfd1346303799006361
I20260929 03:15:39.621415 19769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ab477838f7c4fa793a171b5a12aed9f: Bootstrap starting.
I20260929 03:15:39.624121 19769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ab477838f7c4fa793a171b5a12aed9f: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:39.624354 19772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:39.615629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40698 (local address 127.13.238.190:39761)
0929 03:15:39.616300 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:39.616321 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:39.616366 (+ 45us) server_negotiation.cc:407] Connection header received
0929 03:15:39.616999 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:39.617022 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:39.617549 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:39.617934 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:39.618448 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:39.619858 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:39.620712 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:39.621384 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:39.621571 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:39.622350 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:39.622357 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:39.622372 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:15:39.622870 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:39.622938 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:15:39.622965 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:39.623060 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:39.623156 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 03:15:39.623419 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":108,"threads_started":1}
I20260929 03:15:39.624938 19769 log.cc:824] T 00000000000000000000000000000000 P 6ab477838f7c4fa793a171b5a12aed9f: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:39.627058 19769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ab477838f7c4fa793a171b5a12aed9f: No bootstrap required, opened a new log
I20260929 03:15:39.627161 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:15:39.630620 19769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ab477838f7c4fa793a171b5a12aed9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ab477838f7c4fa793a171b5a12aed9f" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 39761 } }
I20260929 03:15:39.630846 19769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ab477838f7c4fa793a171b5a12aed9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:39.630911 19769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ab477838f7c4fa793a171b5a12aed9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab477838f7c4fa793a171b5a12aed9f, State: Initialized, Role: FOLLOWER
I20260929 03:15:39.631297 19769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ab477838f7c4fa793a171b5a12aed9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ab477838f7c4fa793a171b5a12aed9f" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 39761 } }
I20260929 03:15:39.631419 19769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ab477838f7c4fa793a171b5a12aed9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:15:39.631482 19769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ab477838f7c4fa793a171b5a12aed9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:15:39.631575 19769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ab477838f7c4fa793a171b5a12aed9f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:39.632427 19769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ab477838f7c4fa793a171b5a12aed9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ab477838f7c4fa793a171b5a12aed9f" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 39761 } }
I20260929 03:15:39.632742 19769 leader_election.cc:302] T 00000000000000000000000000000000 P 6ab477838f7c4fa793a171b5a12aed9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ab477838f7c4fa793a171b5a12aed9f; no voters:
I20260929 03:15:39.633059 19769 leader_election.cc:288] T 00000000000000000000000000000000 P 6ab477838f7c4fa793a171b5a12aed9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:15:39.633123 19774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ab477838f7c4fa793a171b5a12aed9f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:39.633332 19774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ab477838f7c4fa793a171b5a12aed9f [term 1 LEADER]: Becoming Leader. State: Replica: 6ab477838f7c4fa793a171b5a12aed9f, State: Running, Role: LEADER
I20260929 03:15:39.633635 19774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ab477838f7c4fa793a171b5a12aed9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ab477838f7c4fa793a171b5a12aed9f" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 39761 } }
I20260929 03:15:39.634224 19769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ab477838f7c4fa793a171b5a12aed9f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:39.635296 19776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ab477838f7c4fa793a171b5a12aed9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ab477838f7c4fa793a171b5a12aed9f. Latest consensus state: current_term: 1 leader_uuid: "6ab477838f7c4fa793a171b5a12aed9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ab477838f7c4fa793a171b5a12aed9f" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 39761 } } }
I20260929 03:15:39.635429 19776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ab477838f7c4fa793a171b5a12aed9f [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:39.635703 19775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ab477838f7c4fa793a171b5a12aed9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ab477838f7c4fa793a171b5a12aed9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ab477838f7c4fa793a171b5a12aed9f" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 39761 } } }
I20260929 03:15:39.635797 19775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ab477838f7c4fa793a171b5a12aed9f [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:39.635886 19783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:15:39.639075 19783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:15:39.644739 19783 catalog_manager.cc:1347] Generated new cluster ID: 8dbb02adc877443ab052f01881e90976
I20260929 03:15:39.644811 19783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:15:39.661697 19783 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:15:39.663003 19783 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:15:39.673065 19783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ab477838f7c4fa793a171b5a12aed9f: Generated new TSK 0
I20260929 03:15:39.673998 19783 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:15:40.161537 19772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:39.687961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40710 (local address 127.13.238.190:39761)
0929 03:15:39.688098 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:39.688103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:39.688224 (+ 121us) server_negotiation.cc:407] Connection header received
0929 03:15:39.688385 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:39.688388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:39.688447 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:39.688573 (+ 126us) server_negotiation.cc:226] Negotiated authn=JWT
0929 03:15:39.688764 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:39.689577 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:39.690450 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:39.690792 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:39.690892 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:39.691155 (+ 263us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 03:15:40.161124 (+469969us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 03:15:40.161414 (+ 290us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40710: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 03:15:40.161610 19772 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40710: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 03:15:40.161741 19798 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:39.687820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:39761 (local address 127.0.0.1:40710)
0929 03:15:39.688142 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:39.688155 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:39.688253 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:39.688562 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:39.688564 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:39.688571 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0929 03:15:39.688673 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:39.688678 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:39.689667 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:39.689669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:39.690340 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:39.690349 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:39.690900 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:39.690903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:39.690969 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:39.690991 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 03:15:40.161376 (+470385us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 03:15:40.161549 (+ 173us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.238.190:39761: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":113,"threads_started":1}
I20260929 03:15:40.162360 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 19702
2026-09-29T03:15:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (759 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (121 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (120 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:15:40 dist-test-slave-nbjg krb5kdc[19810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:15:40 dist-test-slave-nbjg krb5kdc[19810](info): set up 2 sockets
Sep 29 03:15:40 dist-test-slave-nbjg krb5kdc[19810](info): commencing operation
krb5kdc: starting...
W20260929 03:15:41.370499 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:15:41 dist-test-slave-nbjg krb5kdc[19810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:15:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:15:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:15:41.488656 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:42505
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:40043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:42505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:41.597450 19823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:41.597755 19823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:41.597797 19823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:41.601835 19823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:41.601893 19823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:41.601910 19823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:41.601925 19823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:41.601939 19823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:15:41.607432 19823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:40043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:42505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:42505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:41.608441 19823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:41.609431 19823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:41.617439 19829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:41.617452 19831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:41.617504 19823 server_base.cc:1034] running on GCE node
W20260929 03:15:41.617455 19828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:41.618227 19823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:41.619261 19823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:41.620404 19823 hybrid_clock.cc:648] HybridClock initialized: now 1790651741620381 us; error 41 us; skew 500 ppm
Sep 29 03:15:41 dist-test-slave-nbjg krb5kdc[19810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651741, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:41.623013 19823 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:15:41.623857 19823 webserver.cc:460] Webserver started at http://127.13.238.190:34141/ using document root <none> and password file <none>
I20260929 03:15:41.624368 19823 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:41.624430 19823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:41.624629 19823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:41.626618 19823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "3901709edbbb4a2382471cafdc522b02"
format_stamp: "Formatted at 2026-09-29 03:15:41 on dist-test-slave-nbjg"
server_key: "15e1ceebda50a6125f33eaabd22dfe3c"
server_key_iv: "9240f07ada63841cd5ba65aca64fcdbd"
server_key_version: "encryptionkey@0"
I20260929 03:15:41.627136 19823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "3901709edbbb4a2382471cafdc522b02"
format_stamp: "Formatted at 2026-09-29 03:15:41 on dist-test-slave-nbjg"
server_key: "15e1ceebda50a6125f33eaabd22dfe3c"
server_key_iv: "9240f07ada63841cd5ba65aca64fcdbd"
server_key_version: "encryptionkey@0"
I20260929 03:15:41.630653 19823 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:15:41.632934 19838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:41.634148 19823 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:41.634282 19823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "3901709edbbb4a2382471cafdc522b02"
format_stamp: "Formatted at 2026-09-29 03:15:41 on dist-test-slave-nbjg"
server_key: "15e1ceebda50a6125f33eaabd22dfe3c"
server_key_iv: "9240f07ada63841cd5ba65aca64fcdbd"
server_key_version: "encryptionkey@0"
I20260929 03:15:41.634377 19823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:41.644801 19823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:41.648022 19823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:41.648204 19823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:41.654914 19823 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:42505
I20260929 03:15:41.654937 19890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:42505 every 8 connection(s)
I20260929 03:15:41.655849 19823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:15:41.658757 19891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:41.664273 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 19823
I20260929 03:15:41.664386 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:15:41.664372 19891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3901709edbbb4a2382471cafdc522b02: Bootstrap starting.
I20260929 03:15:41.664623 14266 external_mini_cluster.cc:1428] Setting key 3fcbe4c1f07a8c387519c081f807d416
I20260929 03:15:41.666903 19891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3901709edbbb4a2382471cafdc522b02: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:41.667661 19891 log.cc:824] T 00000000000000000000000000000000 P 3901709edbbb4a2382471cafdc522b02: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:41.669561 19891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3901709edbbb4a2382471cafdc522b02: No bootstrap required, opened a new log
Sep 29 03:15:41 dist-test-slave-nbjg krb5kdc[19810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:41.673770 19891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3901709edbbb4a2382471cafdc522b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3901709edbbb4a2382471cafdc522b02" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42505 } }
I20260929 03:15:41.674014 19891 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3901709edbbb4a2382471cafdc522b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:41.674055 19891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3901709edbbb4a2382471cafdc522b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3901709edbbb4a2382471cafdc522b02, State: Initialized, Role: FOLLOWER
I20260929 03:15:41.674525 19891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3901709edbbb4a2382471cafdc522b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3901709edbbb4a2382471cafdc522b02" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42505 } }
I20260929 03:15:41.674646 19891 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3901709edbbb4a2382471cafdc522b02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:15:41.674715 19891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3901709edbbb4a2382471cafdc522b02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:15:41.674814 19891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3901709edbbb4a2382471cafdc522b02 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:41.675678 19891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3901709edbbb4a2382471cafdc522b02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3901709edbbb4a2382471cafdc522b02" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42505 } }
I20260929 03:15:41.676066 19891 leader_election.cc:302] T 00000000000000000000000000000000 P 3901709edbbb4a2382471cafdc522b02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3901709edbbb4a2382471cafdc522b02; no voters:
I20260929 03:15:41.676381 19891 leader_election.cc:288] T 00000000000000000000000000000000 P 3901709edbbb4a2382471cafdc522b02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:15:41.676476 19896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3901709edbbb4a2382471cafdc522b02 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:41.676736 19896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3901709edbbb4a2382471cafdc522b02 [term 1 LEADER]: Becoming Leader. State: Replica: 3901709edbbb4a2382471cafdc522b02, State: Running, Role: LEADER
I20260929 03:15:41.677090 19896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3901709edbbb4a2382471cafdc522b02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3901709edbbb4a2382471cafdc522b02" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42505 } }
I20260929 03:15:41.677670 19891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3901709edbbb4a2382471cafdc522b02 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:41.678511 19896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3901709edbbb4a2382471cafdc522b02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3901709edbbb4a2382471cafdc522b02. Latest consensus state: current_term: 1 leader_uuid: "3901709edbbb4a2382471cafdc522b02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3901709edbbb4a2382471cafdc522b02" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42505 } } }
I20260929 03:15:41.678685 19896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3901709edbbb4a2382471cafdc522b02 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:41.678854 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:41.665843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33200 (local address 127.13.238.190:42505)
0929 03:15:41.666253 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:41.666269 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:41.666304 (+ 35us) server_negotiation.cc:407] Connection header received
0929 03:15:41.666933 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:41.666950 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:41.667305 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:41.667580 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:41.668397 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:41.669373 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:41.670558 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:41.671032 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:41.671146 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:41.673865 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:41.673870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:41.673884 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:41.673909 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:41.675591 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:41.676047 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:41.676052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:41.676057 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:41.676205 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:41.676538 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:41.676542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:41.676544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:41.677069 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:41.677287 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:41.677537 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 03:15:41.677851 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":105,"threads_started":1}
I20260929 03:15:41.679105 19905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:15:41.678908 19897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3901709edbbb4a2382471cafdc522b02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3901709edbbb4a2382471cafdc522b02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3901709edbbb4a2382471cafdc522b02" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42505 } } }
I20260929 03:15:41.679327 19897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3901709edbbb4a2382471cafdc522b02 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:41.682384 19905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:15:41.687597 19905 catalog_manager.cc:1347] Generated new cluster ID: ca36a33c34bc4a45abab3bda9b94a5af
I20260929 03:15:41.687664 19905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:15:41.700517 19905 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:15:41.701586 19905 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:15:41.709172 19905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3901709edbbb4a2382471cafdc522b02: Generated new TSK 0
I20260929 03:15:41.709873 19905 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
I20260929 03:15:41.745337 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:0
--local_ip_for_outbound_sockets=127.13.238.129
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:42505
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:40043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:41.870858 19917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:41.871199 19917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:41.871250 19917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:41.875414 19917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:41.875489 19917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:41.875551 19917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:15:41.881322 19917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:40043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:42505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:41.882439 19917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:41.883481 19917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:41.893004 19925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:41.893065 19917 server_base.cc:1034] running on GCE node
W20260929 03:15:41.892967 19923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:41.892969 19922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:41.893633 19917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:41.894271 19917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:41.895438 19917 hybrid_clock.cc:648] HybridClock initialized: now 1790651741895406 us; error 44 us; skew 500 ppm
Sep 29 03:15:41 dist-test-slave-nbjg krb5kdc[19810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651741, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:41.897933 19917 init.cc:377] Logged in from keytab as kudu/127.13.238.129@KRBTEST.COM (short username kudu)
I20260929 03:15:41.898787 19917 webserver.cc:460] Webserver started at http://127.13.238.129:36465/ using document root <none> and password file <none>
I20260929 03:15:41.899333 19917 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:41.899396 19917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:41.899633 19917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:41.901835 19917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-0/data/instance:
uuid: "99cb15697ed249a59978a164c7ca66f9"
format_stamp: "Formatted at 2026-09-29 03:15:41 on dist-test-slave-nbjg"
server_key: "b71d812243de4d1d727ee99ebb5130ac"
server_key_iv: "9a6e7c235e461510dbf31495d48ab30b"
server_key_version: "encryptionkey@0"
I20260929 03:15:41.902377 19917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance:
uuid: "99cb15697ed249a59978a164c7ca66f9"
format_stamp: "Formatted at 2026-09-29 03:15:41 on dist-test-slave-nbjg"
server_key: "b71d812243de4d1d727ee99ebb5130ac"
server_key_iv: "9a6e7c235e461510dbf31495d48ab30b"
server_key_version: "encryptionkey@0"
I20260929 03:15:41.905877 19917 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:15:41.908241 19932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:41.909422 19917 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:15:41.909528 19917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "99cb15697ed249a59978a164c7ca66f9"
format_stamp: "Formatted at 2026-09-29 03:15:41 on dist-test-slave-nbjg"
server_key: "b71d812243de4d1d727ee99ebb5130ac"
server_key_iv: "9a6e7c235e461510dbf31495d48ab30b"
server_key_version: "encryptionkey@0"
I20260929 03:15:41.909617 19917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:41.925292 19917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:41.927961 19917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:41.928129 19917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:41.928694 19917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:41.929486 19917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:41.929530 19917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:41.929581 19917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:41.929610 19917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:41.937893 19917 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:46185
I20260929 03:15:41.937950 20045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:46185 every 8 connection(s)
I20260929 03:15:41.938859 19917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:15:41.942312 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 19917
I20260929 03:15:41.942420 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance
I20260929 03:15:41.942660 14266 external_mini_cluster.cc:1428] Setting key 9d37ab0869f467375854c3b4917b1a86
Sep 29 03:15:41 dist-test-slave-nbjg krb5kdc[19810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651741, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.238.130@KRBTEST.COM; defaulting to no policy
I20260929 03:15:41.952913 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:41.940645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:38663 (local address 127.13.238.190:42505)
0929 03:15:41.940783 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:41.940788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:41.941598 (+ 810us) server_negotiation.cc:407] Connection header received
0929 03:15:41.943013 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:41.943017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:41.943074 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:41.943150 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:41.944340 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:41.944851 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:41.946405 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:41.946640 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:41.946721 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:41.949496 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:41.949499 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:41.949501 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:41.949524 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:41.950871 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:41.951446 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:41.951448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:41.951449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:41.951515 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:41.951872 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:41.951874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:41.951876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:41.952052 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:41.952127 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:41.952642 (+ 515us) server_negotiation.cc:299] Negotiation successful
0929 03:15:41.952777 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 03:15:41.953749 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:41.941024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42505 (local address 127.13.238.129:38663)
0929 03:15:41.941460 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:41.941502 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:41.942806 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:41.943329 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:41.943337 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:41.943692 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:41.944199 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:41.944213 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:41.944967 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:41.944970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:41.946288 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:41.946300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:41.946786 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:41.946790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:41.946975 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:41.947565 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:41.947586 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:41.949312 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:41.950986 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:41.950994 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:41.951012 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:41.951348 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:41.951586 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:41.951590 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:41.951592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:41.951781 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:41.952144 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:41.952149 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:41.952411 (+ 262us) client_negotiation.cc:769] Sending connection context
0929 03:15:41.952578 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 03:15:41.952879 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.130@KRBTEST.COM" created.
I20260929 03:15:41.955214 20046 heartbeater.cc:347] Connected to a master server at 127.13.238.190:42505
I20260929 03:15:41.955506 20046 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:41.956059 20046 heartbeater.cc:511] Master 127.13.238.190:42505 requested a full tablet report, sending...
I20260929 03:15:41.957888 19855 ts_manager.cc:194] Registered new tserver with Master: 99cb15697ed249a59978a164c7ca66f9 (127.13.238.129:46185)
I20260929 03:15:41.959582 19855 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.129@KRBTEST.COM'} at 127.13.238.129:38663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
I20260929 03:15:41.972808 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:0
--local_ip_for_outbound_sockets=127.13.238.130
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:42505
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:40043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:42.081816 20051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:42.082083 20051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:42.082132 20051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:42.086103 20051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:42.086165 20051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:42.086261 20051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:15:42.091944 20051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:40043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:42505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:42.093029 20051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:42.094048 20051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:15:42.103415 20051 server_base.cc:1034] running on GCE node
W20260929 03:15:42.103520 20059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:42.103453 20056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:42.103372 20057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:42.103938 20051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:42.104535 20051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:42.105686 20051 hybrid_clock.cc:648] HybridClock initialized: now 1790651742105668 us; error 31 us; skew 500 ppm
Sep 29 03:15:42 dist-test-slave-nbjg krb5kdc[19810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651742, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:42.108232 20051 init.cc:377] Logged in from keytab as kudu/127.13.238.130@KRBTEST.COM (short username kudu)
I20260929 03:15:42.109040 20051 webserver.cc:460] Webserver started at http://127.13.238.130:46391/ using document root <none> and password file <none>
I20260929 03:15:42.109545 20051 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:42.109601 20051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:42.109759 20051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:42.111735 20051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-1/data/instance:
uuid: "7e8c67b4719441ba96ae3e31c7847e90"
format_stamp: "Formatted at 2026-09-29 03:15:42 on dist-test-slave-nbjg"
server_key: "bb80f7225d349140bf87b401ac0e38fa"
server_key_iv: "3e497992eab29d53d1cd46dcf52040b1"
server_key_version: "encryptionkey@0"
I20260929 03:15:42.112208 20051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance:
uuid: "7e8c67b4719441ba96ae3e31c7847e90"
format_stamp: "Formatted at 2026-09-29 03:15:42 on dist-test-slave-nbjg"
server_key: "bb80f7225d349140bf87b401ac0e38fa"
server_key_iv: "3e497992eab29d53d1cd46dcf52040b1"
server_key_version: "encryptionkey@0"
I20260929 03:15:42.115829 20051 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 03:15:42.118135 20066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:42.119426 20051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:15:42.119532 20051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "7e8c67b4719441ba96ae3e31c7847e90"
format_stamp: "Formatted at 2026-09-29 03:15:42 on dist-test-slave-nbjg"
server_key: "bb80f7225d349140bf87b401ac0e38fa"
server_key_iv: "3e497992eab29d53d1cd46dcf52040b1"
server_key_version: "encryptionkey@0"
I20260929 03:15:42.119643 20051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:42.140127 20051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:42.142951 20051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:42.143119 20051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:42.143702 20051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:42.144522 20051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:42.144565 20051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:42.144608 20051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:42.144636 20051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:42.152899 20051 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:34177
I20260929 03:15:42.152930 20179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:34177 every 8 connection(s)
I20260929 03:15:42.153837 20051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:15:42.158854 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 20051
I20260929 03:15:42.158947 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance
I20260929 03:15:42.159215 14266 external_mini_cluster.cc:1428] Setting key 91aadd08771ebb6a95ad9e2b862412d0
Sep 29 03:15:42 dist-test-slave-nbjg krb5kdc[19810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651742, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:42.167747 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:42.155676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:51257 (local address 127.13.238.190:42505)
0929 03:15:42.155826 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:42.155830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:42.156689 (+ 859us) server_negotiation.cc:407] Connection header received
0929 03:15:42.158127 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:42.158131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:42.158190 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:42.158307 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:42.159656 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:42.160265 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:42.161494 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:42.161711 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:42.161799 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:42.164200 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:42.164204 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:42.164207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:42.164238 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:42.165742 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:42.166246 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:42.166250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:42.166252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:42.166354 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:42.166652 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:42.166655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:42.166657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:42.166886 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:42.166986 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:42.167441 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 03:15:42.167573 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.13.238.131@KRBTEST.COM; defaulting to no policy
I20260929 03:15:42.168519 20182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:42.156075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42505 (local address 127.13.238.130:51257)
0929 03:15:42.156535 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:42.156579 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:42.157908 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:42.158510 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:42.158517 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:42.158885 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:42.159502 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:42.159518 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:42.160389 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:42.160393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:42.161365 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:42.161376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:42.161848 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:42.161852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:42.162027 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:42.162619 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:42.162636 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:42.164014 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:42.165849 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:42.165854 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:42.165867 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:42.166129 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:42.166435 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:42.166437 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:42.166438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:42.166569 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:42.166975 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:42.166979 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:42.167206 (+ 227us) client_negotiation.cc:769] Sending connection context
0929 03:15:42.167360 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 03:15:42.167613 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":122,"threads_started":1}
I20260929 03:15:42.169661 20180 heartbeater.cc:347] Connected to a master server at 127.13.238.190:42505
I20260929 03:15:42.169901 20180 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:42.170411 20180 heartbeater.cc:511] Master 127.13.238.190:42505 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.131@KRBTEST.COM" created.
I20260929 03:15:42.171530 19855 ts_manager.cc:194] Registered new tserver with Master: 7e8c67b4719441ba96ae3e31c7847e90 (127.13.238.130:34177)
I20260929 03:15:42.172212 19855 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.130@KRBTEST.COM'} at 127.13.238.130:51257
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
I20260929 03:15:42.185510 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:0
--local_ip_for_outbound_sockets=127.13.238.131
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:42505
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:40043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:42.301371 20185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:42.301626 20185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:42.301669 20185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:42.305718 20185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:42.305773 20185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:42.305830 20185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:15:42.311002 20185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:40043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:42505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:42.311991 20185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:42.313014 20185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:42.322268 20190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:42.322345 20185 server_base.cc:1034] running on GCE node
W20260929 03:15:42.322268 20193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:42.322275 20191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:42.322978 20185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:42.323623 20185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:42.324796 20185 hybrid_clock.cc:648] HybridClock initialized: now 1790651742324774 us; error 36 us; skew 500 ppm
Sep 29 03:15:42 dist-test-slave-nbjg krb5kdc[19810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651742, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:42.327329 20185 init.cc:377] Logged in from keytab as kudu/127.13.238.131@KRBTEST.COM (short username kudu)
I20260929 03:15:42.328132 20185 webserver.cc:460] Webserver started at http://127.13.238.131:42313/ using document root <none> and password file <none>
I20260929 03:15:42.328658 20185 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:42.328722 20185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:42.328903 20185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:42.331010 20185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-2/data/instance:
uuid: "eac32d36ac5d45e99910a479cc57f343"
format_stamp: "Formatted at 2026-09-29 03:15:42 on dist-test-slave-nbjg"
server_key: "60f1d05e049a0ea2b174545947f98173"
server_key_iv: "38476a329fe90546f1a8898077e77e30"
server_key_version: "encryptionkey@0"
I20260929 03:15:42.331544 20185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance:
uuid: "eac32d36ac5d45e99910a479cc57f343"
format_stamp: "Formatted at 2026-09-29 03:15:42 on dist-test-slave-nbjg"
server_key: "60f1d05e049a0ea2b174545947f98173"
server_key_iv: "38476a329fe90546f1a8898077e77e30"
server_key_version: "encryptionkey@0"
I20260929 03:15:42.335114 20185 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:15:42.337432 20200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:42.338721 20185 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:42.338826 20185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "eac32d36ac5d45e99910a479cc57f343"
format_stamp: "Formatted at 2026-09-29 03:15:42 on dist-test-slave-nbjg"
server_key: "60f1d05e049a0ea2b174545947f98173"
server_key_iv: "38476a329fe90546f1a8898077e77e30"
server_key_version: "encryptionkey@0"
I20260929 03:15:42.338919 20185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:42.361795 20185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:42.365243 20185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:42.365447 20185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:42.366080 20185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:42.367038 20185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:42.367087 20185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:42.367144 20185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:42.367169 20185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:42.375970 20185 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:40475
I20260929 03:15:42.375993 20313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:40475 every 8 connection(s)
I20260929 03:15:42.376909 20185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:15:42.381388 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 20185
I20260929 03:15:42.381510 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance
I20260929 03:15:42.381786 14266 external_mini_cluster.cc:1428] Setting key 4adbfa742eb024889b5e7e736dd3ab59
Sep 29 03:15:42 dist-test-slave-nbjg krb5kdc[19810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651742, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:42.391064 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:42.378882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:59951 (local address 127.13.238.190:42505)
0929 03:15:42.379022 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:42.379027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:42.380042 (+ 1015us) server_negotiation.cc:407] Connection header received
0929 03:15:42.381397 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:42.381401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:42.381466 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:42.381564 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:42.383035 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:42.383728 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:42.385002 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:42.385239 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:42.385332 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:42.387775 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:42.387778 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:42.387780 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:42.387808 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:42.389037 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:42.389553 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:42.389556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:42.389557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:42.389657 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:42.389991 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:42.389994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:42.389995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:42.390234 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:42.390330 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:42.390825 (+ 495us) server_negotiation.cc:299] Negotiation successful
0929 03:15:42.390935 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:15:42.391930 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:42.379389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42505 (local address 127.13.238.131:59951)
0929 03:15:42.379862 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:42.379924 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:42.381150 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:42.381763 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:42.381771 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:42.382251 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:42.382864 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:42.382883 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:42.383870 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:42.383874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:42.384848 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:42.384859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:42.385374 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:42.385377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:42.385530 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:42.386066 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:42.386083 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:42.387564 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:42.389152 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:42.389158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:42.389172 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:42.389438 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:42.389736 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:42.389738 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:42.389739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:42.389870 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:42.390326 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:42.390332 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:42.390584 (+ 252us) client_negotiation.cc:769] Sending connection context
0929 03:15:42.390747 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 03:15:42.391034 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":118,"threads_started":1}
I20260929 03:15:42.393146 20314 heartbeater.cc:347] Connected to a master server at 127.13.238.190:42505
I20260929 03:15:42.393396 20314 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:42.393885 20314 heartbeater.cc:511] Master 127.13.238.190:42505 requested a full tablet report, sending...
I20260929 03:15:42.395063 19854 ts_manager.cc:194] Registered new tserver with Master: eac32d36ac5d45e99910a479cc57f343 (127.13.238.131:40475)
I20260929 03:15:42.395745 19854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.131@KRBTEST.COM'} at 127.13.238.131:59951
I20260929 03:15:42.395872 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/test-admin.keytab.
Sep 29 03:15:42 dist-test-slave-nbjg krb5kdc[19810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:42.410761 14266 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 03:15:42.413378 20321 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:42.412104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39578 (local address 127.13.238.129:46185)
0929 03:15:42.412394 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:42.412408 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:42.412433 (+ 25us) server_negotiation.cc:407] Connection header received
0929 03:15:42.412482 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:42.412488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:42.412674 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:42.412802 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:42.413204 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39578: BlockingRecv error: recv got EOF from 127.0.0.1:39578 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":90,"threads_started":1}
W20260929 03:15:42.413460 14266 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/krb5cc' of type FILE
Sep 29 03:15:42 dist-test-slave-nbjg krb5kdc[19810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:42.414561 14266 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 03:15:42 dist-test-slave-nbjg krb5kdc[19810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:42.415541 14266 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 03:15:42 dist-test-slave-nbjg krb5kdc[19810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.129@KRBTEST.COM
I20260929 03:15:42.425091 20321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:42.416430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39588 (local address 127.13.238.129:46185)
0929 03:15:42.416552 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:42.416556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:42.416642 (+ 86us) server_negotiation.cc:407] Connection header received
0929 03:15:42.416759 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:42.416762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:42.416801 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:42.416866 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:42.417399 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:42.418172 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:42.419029 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:42.419416 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:42.419510 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:42.421654 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:42.421663 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:42.421677 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:42.421702 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:42.423285 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:42.423766 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:42.423772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:42.423773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:42.423855 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:42.424125 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:42.424128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:42.424129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:42.424383 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:42.424583 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:42.424799 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 03:15:42.424955 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 03:15:42.428650 20321 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:42.427820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39590 (local address 127.13.238.129:46185)
0929 03:15:42.427948 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:42.427951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:42.428012 (+ 61us) server_negotiation.cc:407] Connection header received
0929 03:15:42.428102 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:42.428104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:42.428138 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:42.428196 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:42.428550 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39590: BlockingRecv error: recv got EOF from 127.0.0.1:39590 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 03:15:42.428934 14266 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/krb5cc)
Sep 29 03:15:42 dist-test-slave-nbjg krb5kdc[19810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:42.429841 14266 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 03:15:42 dist-test-slave-nbjg krb5kdc[19810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:42.430648 14266 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 03:15:42 dist-test-slave-nbjg krb5kdc[19810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.129@KRBTEST.COM
I20260929 03:15:42.438341 20321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:42.431401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39604 (local address 127.13.238.129:46185)
0929 03:15:42.431522 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:42.431525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:42.431585 (+ 60us) server_negotiation.cc:407] Connection header received
0929 03:15:42.431723 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:42.431726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:42.431765 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:42.431837 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:42.432223 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:42.432780 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:42.433499 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:42.433721 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:42.433814 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:42.435713 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:42.435716 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:42.435719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:42.435740 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:42.436985 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:42.437347 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:42.437350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:42.437352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:42.437425 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:42.437625 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:42.437627 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:42.437628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:42.437795 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:42.437906 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:42.438077 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 03:15:42.438164 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 03:15:42.440737 20321 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:42.439851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39616 (local address 127.13.238.129:46185)
0929 03:15:42.439946 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:42.439949 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:42.440015 (+ 66us) server_negotiation.cc:407] Connection header received
0929 03:15:42.440092 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:42.440094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:42.440124 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:42.440181 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:42.440636 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39616: BlockingRecv error: recv got EOF from 127.0.0.1:39616 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 03:15:42.440940 14266 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790651713532531-14266-0/krb5kdc/krb5cc' of type FILE
Sep 29 03:15:42 dist-test-slave-nbjg krb5kdc[19810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:42.441861 14266 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 03:15:42 dist-test-slave-nbjg krb5kdc[19810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:42.442732 14266 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 03:15:42 dist-test-slave-nbjg krb5kdc[19810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.129@KRBTEST.COM
I20260929 03:15:42.450351 20321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:42.443503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39632 (local address 127.13.238.129:46185)
0929 03:15:42.443593 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:42.443597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:42.443737 (+ 140us) server_negotiation.cc:407] Connection header received
0929 03:15:42.443828 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:42.443830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:42.443865 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:42.443919 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:42.444361 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:42.444845 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:42.445566 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:42.445770 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:42.445855 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:42.447858 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:42.447861 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:42.447863 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:42.447884 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:42.448955 (+ 1071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:42.449302 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:42.449304 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:42.449306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:42.449374 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:42.449557 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:42.449559 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:42.449560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:42.449705 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:42.449822 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:42.450038 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 03:15:42.450151 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 03:15:42.451776 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 19917
I20260929 03:15:42.458916 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 20051
I20260929 03:15:42.464306 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 20185
I20260929 03:15:42.469210 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 19823
2026-09-29T03:15:42Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2066 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:15:42 dist-test-slave-nbjg krb5kdc[20334](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:15:42 dist-test-slave-nbjg krb5kdc[20334](info): set up 2 sockets
Sep 29 03:15:42 dist-test-slave-nbjg krb5kdc[20334](info): commencing operation
krb5kdc: starting...
W20260929 03:15:43.434464 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:15:43 dist-test-slave-nbjg krb5kdc[20334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:15:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:15:43Z Disabled control of system clock
WARNING: no policy specified for oryx/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.190.keytab.
Entry for principal oryx/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.190.keytab.
I20260929 03:15:43.549674 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:43921
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.190.keytab
--principal=oryx/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:43675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:43921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:43.658062 20347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:43.658401 20347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:43.658460 20347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:43.662750 20347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:43.662834 20347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:43.662863 20347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:43.662884 20347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:43.662910 20347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:15:43.668473 20347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:43675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:43921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.190.keytab
--principal=oryx/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:43921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:43.669597 20347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:43.670675 20347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:43.678854 20355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:43.678918 20347 server_base.cc:1034] running on GCE node
W20260929 03:15:43.678881 20352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:43.678860 20353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:43.679666 20347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:43.680728 20347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:43.681887 20347 hybrid_clock.cc:648] HybridClock initialized: now 1790651743681863 us; error 38 us; skew 500 ppm
Sep 29 03:15:43 dist-test-slave-nbjg krb5kdc[20334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651743, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:43.684614 20347 init.cc:377] Logged in from keytab as oryx/127.13.238.190@KRBTEST.COM (short username oryx)
I20260929 03:15:43.685508 20347 webserver.cc:460] Webserver started at http://127.13.238.190:33635/ using document root <none> and password file <none>
I20260929 03:15:43.686049 20347 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:43.686115 20347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:43.686350 20347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:43.688418 20347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "b4bc38a6af4a493dbba8a0c697b98c77"
format_stamp: "Formatted at 2026-09-29 03:15:43 on dist-test-slave-nbjg"
server_key: "09f2f26fde2bd828d1ec19ae706a4584"
server_key_iv: "29f853dc185d74504ff06687132b2396"
server_key_version: "encryptionkey@0"
I20260929 03:15:43.688961 20347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "b4bc38a6af4a493dbba8a0c697b98c77"
format_stamp: "Formatted at 2026-09-29 03:15:43 on dist-test-slave-nbjg"
server_key: "09f2f26fde2bd828d1ec19ae706a4584"
server_key_iv: "29f853dc185d74504ff06687132b2396"
server_key_version: "encryptionkey@0"
I20260929 03:15:43.692694 20347 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260929 03:15:43.695228 20362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:43.696499 20347 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:43.696626 20347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "b4bc38a6af4a493dbba8a0c697b98c77"
format_stamp: "Formatted at 2026-09-29 03:15:43 on dist-test-slave-nbjg"
server_key: "09f2f26fde2bd828d1ec19ae706a4584"
server_key_iv: "29f853dc185d74504ff06687132b2396"
server_key_version: "encryptionkey@0"
I20260929 03:15:43.696785 20347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:43.721495 20347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:43.724689 20347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:43.724860 20347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:43.731693 20347 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:43921
I20260929 03:15:43.731978 20414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:43921 every 8 connection(s)
I20260929 03:15:43.732687 20347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:15:43.735463 20415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:43.735685 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 20347
I20260929 03:15:43.735798 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:15:43.736088 14266 external_mini_cluster.cc:1428] Setting key 23d8d845f401f202fbc633845a406fae
I20260929 03:15:43.742566 20415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77: Bootstrap starting.
Sep 29 03:15:43 dist-test-slave-nbjg krb5kdc[20334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM
I20260929 03:15:43.745175 20415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:43.745988 20415 log.cc:824] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:43.748174 20415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77: No bootstrap required, opened a new log
I20260929 03:15:43.750909 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:43.737533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33494 (local address 127.13.238.190:43921)
0929 03:15:43.737980 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:43.737994 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:43.738218 (+ 224us) server_negotiation.cc:407] Connection header received
0929 03:15:43.738988 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:43.739006 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:43.739344 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:43.739622 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:43.740427 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:43.741389 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:43.742534 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:43.743065 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:43.743188 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:43.745792 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:43.745798 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:43.745812 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:43.745839 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:43.747841 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:43.748270 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:43.748275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:43.748281 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:43.748414 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:43.748662 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:43.748673 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:43.748674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:43.749157 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:43.749344 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:43.749575 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 03:15:43.749883 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":116,"threads_started":1}
I20260929 03:15:43.752401 20415 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } }
I20260929 03:15:43.752677 20415 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:43.752733 20415 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4bc38a6af4a493dbba8a0c697b98c77, State: Initialized, Role: FOLLOWER
I20260929 03:15:43.753162 20415 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } }
I20260929 03:15:43.753299 20415 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:15:43.753362 20415 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:15:43.753456 20415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:43.754348 20415 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } }
I20260929 03:15:43.754660 20415 leader_election.cc:302] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4bc38a6af4a493dbba8a0c697b98c77; no voters:
I20260929 03:15:43.754937 20415 leader_election.cc:288] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:15:43.755060 20420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:43.755252 20420 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [term 1 LEADER]: Becoming Leader. State: Replica: b4bc38a6af4a493dbba8a0c697b98c77, State: Running, Role: LEADER
I20260929 03:15:43.755525 20420 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } }
I20260929 03:15:43.756068 20415 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:43.757045 20421 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } }
I20260929 03:15:43.757164 20421 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:43.757457 20422 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4bc38a6af4a493dbba8a0c697b98c77. Latest consensus state: current_term: 1 leader_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } }
I20260929 03:15:43.757545 20422 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:43.757665 20429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:15:43.760839 20429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:15:43.766669 20429 catalog_manager.cc:1347] Generated new cluster ID: e30d83aca1b1425e8a563e680cc2740b
I20260929 03:15:43.766742 20429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:15:43.773077 20429 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:15:43.773995 20429 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:15:43.778796 20429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77: Generated new TSK 0
I20260929 03:15:43.779460 20429 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.13.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.129.keytab.
Entry for principal oryx/127.13.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.129.keytab.
I20260929 03:15:43.812762 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:0
--local_ip_for_outbound_sockets=127.13.238.129
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43921
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.129.keytab
--principal=oryx/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:43675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:43.944927 20441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:43.945178 20441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:43.945220 20441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:43.949265 20441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:43.949324 20441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:43.949398 20441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:15:43.954658 20441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:43675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.129.keytab
--principal=oryx/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:43.955782 20441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:43.956974 20441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:43.966754 20447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:43.966773 20446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:43.966792 20449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:43.967118 20441 server_base.cc:1034] running on GCE node
I20260929 03:15:43.967551 20441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:43.968220 20441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:43.969370 20441 hybrid_clock.cc:648] HybridClock initialized: now 1790651743969351 us; error 32 us; skew 500 ppm
Sep 29 03:15:43 dist-test-slave-nbjg krb5kdc[20334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651743, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:43.971971 20441 init.cc:377] Logged in from keytab as oryx/127.13.238.129@KRBTEST.COM (short username oryx)
I20260929 03:15:43.972783 20441 webserver.cc:460] Webserver started at http://127.13.238.129:46285/ using document root <none> and password file <none>
I20260929 03:15:43.973312 20441 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:43.973371 20441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:43.973547 20441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:43.975620 20441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/data/instance:
uuid: "4350ad5a78764d1db39c247d759ce213"
format_stamp: "Formatted at 2026-09-29 03:15:43 on dist-test-slave-nbjg"
server_key: "b0c40ca6edd75fdba3d3c69c12e00dcf"
server_key_iv: "0e732300ba6d63961a2819f9c13b586a"
server_key_version: "encryptionkey@0"
I20260929 03:15:43.976150 20441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance:
uuid: "4350ad5a78764d1db39c247d759ce213"
format_stamp: "Formatted at 2026-09-29 03:15:43 on dist-test-slave-nbjg"
server_key: "b0c40ca6edd75fdba3d3c69c12e00dcf"
server_key_iv: "0e732300ba6d63961a2819f9c13b586a"
server_key_version: "encryptionkey@0"
I20260929 03:15:43.979934 20441 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 03:15:43.982381 20456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:43.983597 20441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 03:15:43.983705 20441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "4350ad5a78764d1db39c247d759ce213"
format_stamp: "Formatted at 2026-09-29 03:15:43 on dist-test-slave-nbjg"
server_key: "b0c40ca6edd75fdba3d3c69c12e00dcf"
server_key_iv: "0e732300ba6d63961a2819f9c13b586a"
server_key_version: "encryptionkey@0"
I20260929 03:15:43.983801 20441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:44.014500 20441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:44.017445 20441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:44.017620 20441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:44.018280 20441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:44.019148 20441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:44.019196 20441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:44.019250 20441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:44.019276 20441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:44.027949 20441 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:39585
I20260929 03:15:44.027971 20569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:39585 every 8 connection(s)
I20260929 03:15:44.028810 20441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:15:44.029829 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 20441
I20260929 03:15:44.029960 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance
I20260929 03:15:44.030303 14266 external_mini_cluster.cc:1428] Setting key 9aee268cc7fd75f189f9ecb638ca27e5
Sep 29 03:15:44 dist-test-slave-nbjg krb5kdc[20334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651743, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.13.238.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.238.130@KRBTEST.COM" created.
I20260929 03:15:44.043574 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:44.031482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:55933 (local address 127.13.238.190:43921)
0929 03:15:44.031610 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:44.031614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:44.031754 (+ 140us) server_negotiation.cc:407] Connection header received
0929 03:15:44.033160 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:44.033163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:44.033220 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:44.033290 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:44.034614 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.035149 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.036864 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.037102 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.037200 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:44.040025 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:44.040028 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:44.040030 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:44.040056 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:44.041397 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:44.041874 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:44.041877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:44.041878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:44.041968 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:44.042407 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:44.042410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:44.042411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:44.042636 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:44.042740 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:44.043258 (+ 518us) server_negotiation.cc:299] Negotiation successful
0929 03:15:44.043399 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 03:15:44.044469 20572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:44.031118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.13.238.129:55933)
0929 03:15:44.031609 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:44.031661 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:44.032946 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:44.033476 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:44.033484 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:44.033832 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:44.034465 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:44.034485 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.035289 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.035293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:44.036716 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:44.036730 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.037272 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.037277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:44.037485 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:44.038088 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:44.038110 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:44.039847 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:44.041506 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:44.041512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:44.041525 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:44.041764 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:44.042088 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:44.042091 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:44.042092 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:44.042313 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:44.042746 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:44.042752 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:44.043005 (+ 253us) client_negotiation.cc:769] Sending connection context
0929 03:15:44.043161 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 03:15:44.043430 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":110,"threads_started":1}
I20260929 03:15:44.045918 20570 heartbeater.cc:347] Connected to a master server at 127.13.238.190:43921
I20260929 03:15:44.046178 20570 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:44.046767 20570 heartbeater.cc:511] Master 127.13.238.190:43921 requested a full tablet report, sending...
I20260929 03:15:44.048297 20379 ts_manager.cc:194] Registered new tserver with Master: 4350ad5a78764d1db39c247d759ce213 (127.13.238.129:39585)
I20260929 03:15:44.049423 20379 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.238.129@KRBTEST.COM'} at 127.13.238.129:55933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.130.keytab.
Entry for principal oryx/127.13.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.130.keytab.
I20260929 03:15:44.057262 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:0
--local_ip_for_outbound_sockets=127.13.238.130
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43921
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.130.keytab
--principal=oryx/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:43675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:44.167821 20575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:44.168095 20575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:44.168157 20575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:44.172199 20575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:44.172257 20575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:44.172336 20575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:15:44.177595 20575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:43675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.130.keytab
--principal=oryx/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:44.178694 20575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:44.179708 20575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:44.188995 20580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:44.189004 20583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:44.189005 20581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:44.189105 20575 server_base.cc:1034] running on GCE node
I20260929 03:15:44.189646 20575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:44.190286 20575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:44.191437 20575 hybrid_clock.cc:648] HybridClock initialized: now 1790651744191417 us; error 28 us; skew 500 ppm
Sep 29 03:15:44 dist-test-slave-nbjg krb5kdc[20334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651744, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:44.194023 20575 init.cc:377] Logged in from keytab as oryx/127.13.238.130@KRBTEST.COM (short username oryx)
I20260929 03:15:44.194896 20575 webserver.cc:460] Webserver started at http://127.13.238.130:35891/ using document root <none> and password file <none>
I20260929 03:15:44.195400 20575 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:44.195453 20575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:44.195613 20575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:44.197547 20575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/data/instance:
uuid: "2ae2e4cfa8764352a502fbe7ff3009c4"
format_stamp: "Formatted at 2026-09-29 03:15:44 on dist-test-slave-nbjg"
server_key: "87147124e12f965d2fc7f42df1e1b63c"
server_key_iv: "c2b306599571989f2265c8e5b7ad94ee"
server_key_version: "encryptionkey@0"
I20260929 03:15:44.198050 20575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance:
uuid: "2ae2e4cfa8764352a502fbe7ff3009c4"
format_stamp: "Formatted at 2026-09-29 03:15:44 on dist-test-slave-nbjg"
server_key: "87147124e12f965d2fc7f42df1e1b63c"
server_key_iv: "c2b306599571989f2265c8e5b7ad94ee"
server_key_version: "encryptionkey@0"
I20260929 03:15:44.201637 20575 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260929 03:15:44.203918 20590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:44.205053 20575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 03:15:44.205144 20575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "2ae2e4cfa8764352a502fbe7ff3009c4"
format_stamp: "Formatted at 2026-09-29 03:15:44 on dist-test-slave-nbjg"
server_key: "87147124e12f965d2fc7f42df1e1b63c"
server_key_iv: "c2b306599571989f2265c8e5b7ad94ee"
server_key_version: "encryptionkey@0"
I20260929 03:15:44.205230 20575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:44.219739 20575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:44.223114 20575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:44.223302 20575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:44.223958 20575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:44.224874 20575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:44.224922 20575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:44.224963 20575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:44.224992 20575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:44.232889 20575 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:35993
I20260929 03:15:44.232950 20703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:35993 every 8 connection(s)
I20260929 03:15:44.233758 20575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:15:44 dist-test-slave-nbjg krb5kdc[20334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651744, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM
I20260929 03:15:44.243220 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 20575
I20260929 03:15:44.243326 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance
I20260929 03:15:44.243634 14266 external_mini_cluster.cc:1428] Setting key ad3e5b0ecb05bc7705edde07dbcb9c16
I20260929 03:15:44.246323 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:44.235430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:53441 (local address 127.13.238.190:43921)
0929 03:15:44.235562 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:44.235566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:44.236369 (+ 803us) server_negotiation.cc:407] Connection header received
0929 03:15:44.237599 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:44.237602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:44.237658 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:44.237762 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:44.238901 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.239415 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.240483 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.240705 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.240786 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:44.242995 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:44.242998 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:44.243000 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:44.243026 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:44.244397 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:44.244852 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:44.244855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:44.244856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:44.244934 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:44.245242 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:44.245246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:44.245247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:44.245424 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:44.245509 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:44.245985 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 03:15:44.246114 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:15:44.247216 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:44.235789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.13.238.130:53441)
0929 03:15:44.236220 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:44.236262 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:44.237416 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:44.237912 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:44.237919 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:44.238303 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:44.238766 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:44.238788 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.239503 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.239506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:44.240365 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:44.240373 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.240820 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.240822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:44.240965 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:44.241437 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:44.241451 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:44.242829 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:44.244500 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:44.244505 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:44.244521 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:44.244760 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:44.245023 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:44.245026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:44.245027 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:44.245154 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:44.245518 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:44.245524 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:44.245762 (+ 238us) client_negotiation.cc:769] Sending connection context
0929 03:15:44.245927 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 03:15:44.246266 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":109,"threads_started":1}
I20260929 03:15:44.248260 20704 heartbeater.cc:347] Connected to a master server at 127.13.238.190:43921
I20260929 03:15:44.248485 20704 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:44.248948 20704 heartbeater.cc:511] Master 127.13.238.190:43921 requested a full tablet report, sending...
I20260929 03:15:44.250056 20379 ts_manager.cc:194] Registered new tserver with Master: 2ae2e4cfa8764352a502fbe7ff3009c4 (127.13.238.130:35993)
I20260929 03:15:44.250675 20379 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.238.130@KRBTEST.COM'} at 127.13.238.130:53441
WARNING: no policy specified for oryx/127.13.238.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.131.keytab.
Entry for principal oryx/127.13.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.131.keytab.
I20260929 03:15:44.271543 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:0
--local_ip_for_outbound_sockets=127.13.238.131
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43921
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.131.keytab
--principal=oryx/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:43675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:44.380789 20709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:44.381055 20709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:44.381098 20709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:44.385099 20709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:44.385162 20709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:44.385257 20709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:15:44.390609 20709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:43675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.131.keytab
--principal=oryx/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:44.391588 20709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:44.392549 20709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:44.401855 20714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:44.401854 20715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:44.401849 20717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:44.401950 20709 server_base.cc:1034] running on GCE node
I20260929 03:15:44.402484 20709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:44.403126 20709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:44.404292 20709 hybrid_clock.cc:648] HybridClock initialized: now 1790651744404256 us; error 32 us; skew 500 ppm
Sep 29 03:15:44 dist-test-slave-nbjg krb5kdc[20334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651744, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:44.407441 20709 init.cc:377] Logged in from keytab as oryx/127.13.238.131@KRBTEST.COM (short username oryx)
I20260929 03:15:44.408355 20709 webserver.cc:460] Webserver started at http://127.13.238.131:32795/ using document root <none> and password file <none>
I20260929 03:15:44.408962 20709 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:44.409025 20709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:44.409219 20709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:44.411342 20709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/data/instance:
uuid: "bac6c1b9f519437f82e706c280a0f13f"
format_stamp: "Formatted at 2026-09-29 03:15:44 on dist-test-slave-nbjg"
server_key: "85ca053e2060f3a9afecac635d674b91"
server_key_iv: "351a3813d3929b7f1e59ffda4810ec0c"
server_key_version: "encryptionkey@0"
I20260929 03:15:44.411886 20709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance:
uuid: "bac6c1b9f519437f82e706c280a0f13f"
format_stamp: "Formatted at 2026-09-29 03:15:44 on dist-test-slave-nbjg"
server_key: "85ca053e2060f3a9afecac635d674b91"
server_key_iv: "351a3813d3929b7f1e59ffda4810ec0c"
server_key_version: "encryptionkey@0"
I20260929 03:15:44.415520 20709 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:15:44.417882 20724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:44.419080 20709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:44.419193 20709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "bac6c1b9f519437f82e706c280a0f13f"
format_stamp: "Formatted at 2026-09-29 03:15:44 on dist-test-slave-nbjg"
server_key: "85ca053e2060f3a9afecac635d674b91"
server_key_iv: "351a3813d3929b7f1e59ffda4810ec0c"
server_key_version: "encryptionkey@0"
I20260929 03:15:44.419284 20709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:44.433166 20709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:44.435889 20709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:44.436045 20709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:44.436607 20709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:44.437438 20709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:44.437484 20709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:44.437536 20709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:44.437563 20709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:44.445950 20709 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:38001
I20260929 03:15:44.445971 20837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:38001 every 8 connection(s)
I20260929 03:15:44.446857 20709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:15:44.447141 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 20709
I20260929 03:15:44.447252 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance
I20260929 03:15:44.447599 14266 external_mini_cluster.cc:1428] Setting key afe02f140a4ad98385c68649774d61bb
Sep 29 03:15:44 dist-test-slave-nbjg krb5kdc[20334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651744, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM
I20260929 03:15:44.459962 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:44.448621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:51181 (local address 127.13.238.190:43921)
0929 03:15:44.448785 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:44.448790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:44.449695 (+ 905us) server_negotiation.cc:407] Connection header received
0929 03:15:44.451050 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:44.451053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:44.451108 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:44.451217 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:44.452420 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.452993 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.454164 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.454441 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.454535 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:44.456901 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:44.456905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:44.456907 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:44.456933 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:44.458187 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:44.458678 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:44.458681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:44.458682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:44.458767 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:44.459037 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:44.459039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:44.459041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:44.459208 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:44.459281 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:44.459714 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 03:15:44.459813 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 03:15:44.460776 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:44.449121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.13.238.131:51181)
0929 03:15:44.449558 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:44.449599 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:44.450837 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:44.451425 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:44.451433 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:44.451789 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:44.452273 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:44.452288 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.453116 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.453120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:44.454043 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:44.454054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.454589 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.454593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:44.454776 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:44.455273 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:44.455289 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:44.456721 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:44.458329 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:44.458335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:44.458349 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:44.458600 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:44.458835 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:44.458837 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:44.458838 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:44.458970 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:44.459284 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:44.459288 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:44.459525 (+ 237us) client_negotiation.cc:769] Sending connection context
0929 03:15:44.459677 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 03:15:44.459936 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":107,"threads_started":1}
I20260929 03:15:44.461987 20838 heartbeater.cc:347] Connected to a master server at 127.13.238.190:43921
I20260929 03:15:44.462253 20838 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:44.462795 20838 heartbeater.cc:511] Master 127.13.238.190:43921 requested a full tablet report, sending...
I20260929 03:15:44.463819 20379 ts_manager.cc:194] Registered new tserver with Master: bac6c1b9f519437f82e706c280a0f13f (127.13.238.131:38001)
I20260929 03:15:44.464372 20379 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.238.131@KRBTEST.COM'} at 127.13.238.131:51181
I20260929 03:15:44.474855 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.13.238.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.238.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.238.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.189.keytab.
Entry for principal oryx/127.13.238.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.189.keytab.
I20260929 03:15:44.501037 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.189:46533
--webserver_interface=127.13.238.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.189.keytab
--principal=oryx/127.13.238.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:43675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:43921,127.13.238.189:46533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:44.619279 20843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:44.619709 20843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:44.619789 20843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:44.625579 20843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:44.625664 20843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:44.625757 20843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:44.625788 20843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:44.625818 20843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:15:44.633543 20843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:43675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:43921,127.13.238.189:46533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.189.keytab
--principal=oryx/127.13.238.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.189:46533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.238.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
W20260929 03:15:44.633862 20843 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.13.238.190:43921,127.13.238.189:46533'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 03:15:44.635177 20843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:44.636636 20843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:44.647069 20851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:44.647061 20849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:44.647061 20848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:44.647248 20843 server_base.cc:1034] running on GCE node
I20260929 03:15:44.647975 20843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:44.649484 20843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:44.650647 20843 hybrid_clock.cc:648] HybridClock initialized: now 1790651744650644 us; error 55 us; skew 500 ppm
Sep 29 03:15:44 dist-test-slave-nbjg krb5kdc[20334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651744, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:44.653357 20843 init.cc:377] Logged in from keytab as oryx/127.13.238.189@KRBTEST.COM (short username oryx)
I20260929 03:15:44.654328 20843 webserver.cc:460] Webserver started at http://127.13.238.189:41187/ using document root <none> and password file <none>
I20260929 03:15:44.654881 20843 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:44.654947 20843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:44.655138 20843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:44.657342 20843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-1/data/instance:
uuid: "12996e1d57454853ba637b4f8660e902"
format_stamp: "Formatted at 2026-09-29 03:15:44 on dist-test-slave-nbjg"
server_key: "1d957f90c0af847ce4e51d004764a14c"
server_key_iv: "1c70137ea9ddb8aa44d0094ff0cb6685"
server_key_version: "encryptionkey@0"
I20260929 03:15:44.657946 20843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-1/wal/instance:
uuid: "12996e1d57454853ba637b4f8660e902"
format_stamp: "Formatted at 2026-09-29 03:15:44 on dist-test-slave-nbjg"
server_key: "1d957f90c0af847ce4e51d004764a14c"
server_key_iv: "1c70137ea9ddb8aa44d0094ff0cb6685"
server_key_version: "encryptionkey@0"
I20260929 03:15:44.662070 20843 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260929 03:15:44.664798 20858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:44.666109 20843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:15:44.666266 20843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-1/wal
uuid: "12996e1d57454853ba637b4f8660e902"
format_stamp: "Formatted at 2026-09-29 03:15:44 on dist-test-slave-nbjg"
server_key: "1d957f90c0af847ce4e51d004764a14c"
server_key_iv: "1c70137ea9ddb8aa44d0094ff0cb6685"
server_key_version: "encryptionkey@0"
I20260929 03:15:44.666391 20843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:44.693930 20843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:44.696909 20843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:44.697096 20843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:44.704191 20843 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.189:46533
I20260929 03:15:44.704207 20910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.189:46533 every 8 connection(s)
I20260929 03:15:44.705168 20843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-1/data/info.pb
I20260929 03:15:44.707506 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 20843
I20260929 03:15:44.708458 20911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:44.713378 20911 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:15:44 dist-test-slave-nbjg krb5kdc[20334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.238.189@KRBTEST.COM
Sep 29 03:15:44 dist-test-slave-nbjg krb5kdc[20334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651744, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.189@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM
I20260929 03:15:44.722883 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:44.709086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34458 (local address 127.13.238.189:46533)
0929 03:15:44.709543 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:44.709562 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:44.709606 (+ 44us) server_negotiation.cc:407] Connection header received
0929 03:15:44.710493 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:44.710521 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:44.710991 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:44.711411 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:44.712140 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.713636 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.714529 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.715168 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.715321 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:44.717936 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:44.717943 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:44.717955 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:44.717986 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:44.719838 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:44.720316 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:44.720322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:44.720328 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:44.720467 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:44.720698 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:44.720701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:44.720703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:44.721147 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:44.721339 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:44.721631 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 03:15:44.721905 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":125,"threads_started":1}
I20260929 03:15:44.724835 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:44.715463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33504 (local address 127.13.238.190:43921)
0929 03:15:44.715594 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:44.715599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:44.715998 (+ 399us) server_negotiation.cc:407] Connection header received
0929 03:15:44.716124 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:44.716127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:44.716175 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:44.716241 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:44.716929 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.717494 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.718752 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.718975 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.719067 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:44.721383 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:44.721386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:44.721389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:44.721416 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:44.722909 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:44.723427 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:44.723431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:44.723432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:44.723523 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:44.723749 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:44.723752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:44.723754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:44.724149 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:44.724253 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:44.724592 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 03:15:44.724694 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 03:15:44.724890 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:44.715400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.0.0.1:33504)
0929 03:15:44.715857 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:44.715913 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:44.716033 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:44.716304 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:44.716310 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:44.716668 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:44.716791 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:44.716797 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.717622 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.717624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:44.718564 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:44.718575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.719084 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.719088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:44.719193 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:44.719619 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:44.719631 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:44.721234 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:44.723033 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:44.723043 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:44.723051 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:44.723323 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:44.723607 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:44.723609 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:44.723610 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:44.723680 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:44.724280 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:44.724286 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:44.724400 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 03:15:44.724540 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 03:15:44.724705 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":161,"threads_started":1}
I20260929 03:15:44.726264 20911 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:15:44 dist-test-slave-nbjg krb5kdc[20334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651744, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.189@KRBTEST.COM for oryx/127.13.238.189@KRBTEST.COM
I20260929 03:15:44.733964 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:44.726692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:46533 (local address 127.0.0.1:34468)
0929 03:15:44.726800 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:44.726813 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:44.726872 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:44.727038 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:44.727040 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:44.727156 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:44.727242 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:44.727246 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.728514 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.728517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:44.729011 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:44.729018 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.729464 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.729466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:44.729529 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:44.729931 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:44.729941 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:44.731212 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:44.732618 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:44.732621 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:44.732623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:44.732844 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:44.733052 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:44.733055 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:44.733057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:44.733141 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:44.733470 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:44.733473 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:44.733563 (+ 90us) client_negotiation.cc:769] Sending connection context
0929 03:15:44.733662 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 03:15:44.733793 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 03:15:44.733968 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:44.726746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34468 (local address 127.13.238.189:46533)
0929 03:15:44.726851 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:44.726855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:44.726868 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:15:44.726930 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:44.726932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:44.726964 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:44.727022 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:44.727351 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.728404 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.729140 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.729358 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.729464 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:44.731331 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:44.731334 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:44.731336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:44.731355 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:44.732521 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:44.732922 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:44.732925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:44.732926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:44.732985 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:44.733223 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:44.733224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:44.733225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:44.733387 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:44.733460 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:44.733718 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 03:15:44.733844 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":36}
I20260929 03:15:44.737413 20911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902: Bootstrap starting.
I20260929 03:15:44.740192 20911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:44.741030 20911 log.cc:824] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:44.743170 20911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902: No bootstrap required, opened a new log
I20260929 03:15:44.746711 20911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } }
I20260929 03:15:44.746919 20911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:44.746963 20911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12996e1d57454853ba637b4f8660e902, State: Initialized, Role: FOLLOWER
I20260929 03:15:44.747375 20911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } }
I20260929 03:15:44.747797 20917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } } }
I20260929 03:15:44.747918 20917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:15:44.748564 20911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 03:15:44.753099 20928 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:15:44.753183 20928 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:15:44.755432 20843 master_runner.cc:428] Detected that this master 12996e1d57454853ba637b4f8660e902 is joining an existing cluster
I20260929 03:15:44.755501 20843 master_runner.cc:432] Initiating AddMaster RPC to add 127.13.238.189:46533
I20260929 03:15:44.765726 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:44.758265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34484 (local address 127.13.238.189:46533)
0929 03:15:44.758357 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:44.758362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:44.758667 (+ 305us) server_negotiation.cc:407] Connection header received
0929 03:15:44.758800 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:44.758803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:44.758843 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:44.758916 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:44.759385 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.760086 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.761002 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.761256 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.761348 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:44.762660 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:44.762663 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:44.762665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:44.762685 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:44.764251 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:44.764673 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:44.764676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:44.764678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:44.764747 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:44.764993 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:44.764996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:44.764997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:44.765163 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:44.765286 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:44.765483 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 03:15:44.765594 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":29}
I20260929 03:15:44.766301 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:44.758303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:46533 (local address 127.0.0.1:34484)
0929 03:15:44.758572 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:44.758585 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:44.758707 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:44.758943 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:44.758946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:44.759146 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:44.759277 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:44.759284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.760193 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.760196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:44.760899 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:44.760908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.761358 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.761361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:44.761465 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:44.762081 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:44.762097 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:44.762549 (+ 452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:44.764352 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:44.764355 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:44.764358 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:44.764578 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:44.764839 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:44.764842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:44.764844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:44.764914 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:44.765258 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:44.765260 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:44.765336 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 03:15:44.765964 (+ 628us) client_negotiation.cc:240] Negotiation successful
0929 03:15:44.766089 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":85,"thread_start_us":73,"threads_started":1}
I20260929 03:15:44.766592 20934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:44.757795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.0.0.1:33510)
0929 03:15:44.758086 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:44.758101 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:44.758220 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:44.758668 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:44.758671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:44.758921 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:44.759129 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:44.759135 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.760147 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.760150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:44.761178 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:44.761189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.761767 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.761771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:44.761859 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:44.762711 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:44.762726 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:44.763112 (+ 386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:44.765035 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:44.765038 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:44.765040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:44.765303 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:44.765579 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:44.765582 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:44.765584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:44.765674 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:44.766113 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:44.766116 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:44.766225 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 03:15:44.766325 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 03:15:44.766449 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":83,"thread_start_us":85,"threads_started":1}
I20260929 03:15:44.766713 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:44.757917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33510 (local address 127.13.238.190:43921)
0929 03:15:44.758038 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:44.758043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:44.758169 (+ 126us) server_negotiation.cc:407] Connection header received
0929 03:15:44.758437 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:44.758441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:44.758497 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:44.758543 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:44.759234 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.760036 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.761428 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.761671 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.761775 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:44.763240 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:44.763243 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:44.763246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:44.763277 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:44.764937 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:44.765405 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:44.765409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:44.765411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:44.765498 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:44.765765 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:44.765768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:44.765770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:44.765987 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:44.766074 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:44.766422 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 03:15:44.766542 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Sep 29 03:15:44 dist-test-slave-nbjg krb5kdc[20334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651743, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.190@KRBTEST.COM for oryx/127.13.238.189@KRBTEST.COM
I20260929 03:15:44.781248 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:44.771219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34488 (local address 127.13.238.189:46533)
0929 03:15:44.771335 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:44.771339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:44.771681 (+ 342us) server_negotiation.cc:407] Connection header received
0929 03:15:44.771867 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:44.771870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:44.771922 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:44.772002 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:44.772620 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.773385 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.774336 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.774772 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.774869 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:44.777448 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:44.777452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:44.777454 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:44.777479 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:44.779201 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:44.779709 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:44.779712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:44.779714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:44.779799 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:44.780169 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:44.780172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:44.780174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:44.780449 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:44.780545 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:44.780964 (+ 419us) server_negotiation.cc:299] Negotiation successful
0929 03:15:44.781087 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:15:44.781320 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:44.771184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:46533 (local address 127.0.0.1:34488)
0929 03:15:44.771560 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:44.771612 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:44.771744 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:44.772041 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:44.772046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:44.772410 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:44.772524 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:44.772530 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.773510 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.773513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:44.774189 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:44.774232 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.774889 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.774892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:44.774979 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:44.775521 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:44.775533 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:44.777299 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:44.779308 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:44.779318 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:44.779325 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:44.779614 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:44.779880 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:44.779883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:44.779885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:44.779974 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:44.780562 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:44.780570 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:44.780725 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 03:15:44.780898 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 03:15:44.781127 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":101,"threads_started":1}
I20260929 03:15:44.783140 20379 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.13.238.189:46533: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "12996e1d57454853ba637b4f8660e902"
member_type: NON_VOTER
last_known_addr {
host: "127.13.238.189"
port: 46533
}
attrs {
promote: true
}
}
dest_uuid: "b4bc38a6af4a493dbba8a0c697b98c77"
cas_config_opid_index: -1
I20260929 03:15:44.783878 20379 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: NON_VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.13.238.188@KRBTEST.COM; defaulting to no policy
I20260929 03:15:44.786098 20939 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 12996e1d57454853ba637b4f8660e902 (127.13.238.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: NON_VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: true } } }
I20260929 03:15:44.786602 20885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:44.787885 20939 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.13.238.189:46533
I20260929 03:15:44.788130 20885 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [term 1 FOLLOWER]: Refusing update from remote peer b4bc38a6af4a493dbba8a0c697b98c77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 03:15:44.788106 20939 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: NON_VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: true } } }
I20260929 03:15:44.788214 20939 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:44.788403 20843 master.cc:537] Master@127.13.238.189:46533 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.238.188@KRBTEST.COM" created.
I20260929 03:15:44.789507 20940 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: NON_VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 03:15:44.791570 20364 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 03:15:44.792551 20364 consensus_peers.cc:487] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 -> Peer 12996e1d57454853ba637b4f8660e902 (127.13.238.189:46533): Couldn't send request to peer 12996e1d57454853ba637b4f8660e902. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 03:15:44.793970 20843 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:15:44.794293 20843 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:44.794416 20843 tablet_replica.cc:330] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902: stopping tablet replica
I20260929 03:15:44.798519 20843 master.cc:552] Master@127.13.238.189:46533 shutdown complete.
I20260929 03:15:44.798559 20843 master_runner.cc:305] Clearing existing system tablet
I20260929 03:15:44.798794 20843 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:44.798847 20843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:44.799850 20843 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:44.801085 20945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:44.801357 20843 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:44.801460 20843 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:15:44.801497 20843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-1/wal
uuid: "12996e1d57454853ba637b4f8660e902"
format_stamp: "Formatted at 2026-09-29 03:15:44 on dist-test-slave-nbjg"
server_key: "1d957f90c0af847ce4e51d004764a14c"
server_key_iv: "1c70137ea9ddb8aa44d0094ff0cb6685"
server_key_version: "encryptionkey@0"
I20260929 03:15:44.803437 20843 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:44.805670 20843 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 03:15:44.805763 20843 log.cc:1196] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 03:15:44.806378 20843 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902: Deleting consensus metadata
I20260929 03:15:44.806543 20843 master_runner.cc:315] Copying system tablet from 127.13.238.190:43921
I20260929 03:15:44.807654 20843 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902: tablet copy: Beginning tablet copy session from remote peer at address 127.13.238.190:43921
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.238.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.188.keytab.
Entry for principal oryx/127.13.238.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.188.keytab.
I20260929 03:15:44.809594 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.188:44881
--webserver_interface=127.13.238.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.188.keytab
--principal=oryx/127.13.238.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:43675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:43921,127.13.238.189:46533,127.13.238.188:44881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:15:44.816287 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:44.808289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33524 (local address 127.13.238.190:43921)
0929 03:15:44.808416 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:44.808421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:44.809035 (+ 614us) server_negotiation.cc:407] Connection header received
0929 03:15:44.809094 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:44.809097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:44.809154 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:44.809245 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:44.809816 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.810673 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.811897 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.812175 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.812269 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:44.813359 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:44.813364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:44.813366 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:44.813395 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:44.814741 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:44.815138 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:44.815142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:44.815144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:44.815224 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:44.815479 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:44.815483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:44.815485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:44.815672 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:44.815755 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:44.816007 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 03:15:44.816120 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 03:15:44.816295 20951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:44.808205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.0.0.1:33524)
0929 03:15:44.808495 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:44.808509 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:44.808631 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:44.809288 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:44.809291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:44.809565 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:44.809684 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:44.809692 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.810816 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.810820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:44.811761 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:44.811773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:44.812302 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:44.812306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:44.812403 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:44.812883 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:44.812897 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:44.813216 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:44.814844 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:44.814848 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:44.814850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:44.815049 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:44.815302 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:44.815306 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:44.815308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:44.815399 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:44.815758 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:44.815761 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:44.815857 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 03:15:44.815940 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 03:15:44.816088 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":88,"threads_started":1}
I20260929 03:15:44.817183 20399 tablet_copy_service.cc:140] P b4bc38a6af4a493dbba8a0c697b98c77: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 12996e1d57454853ba637b4f8660e902 ({username='oryx', principal='oryx/127.13.238.189@KRBTEST.COM'} at 127.0.0.1:33524)
I20260929 03:15:44.817356 20399 tablet_copy_service.cc:161] P b4bc38a6af4a493dbba8a0c697b98c77: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 12996e1d57454853ba637b4f8660e902 at {username='oryx', principal='oryx/127.13.238.189@KRBTEST.COM'} at 127.0.0.1:33524: session id = 12996e1d57454853ba637b4f8660e902-00000000000000000000000000000000
I20260929 03:15:44.819087 20399 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 03:15:44.820629 20843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 03:15:44.822629 20843 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902: tablet copy: Starting download of 0 data blocks...
I20260929 03:15:44.822800 20843 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902: tablet copy: Starting download of 1 WAL segments...
I20260929 03:15:44.824476 20843 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 03:15:44.825964 20399 tablet_copy_service.cc:342] P b4bc38a6af4a493dbba8a0c697b98c77: Request end of tablet copy session 12996e1d57454853ba637b4f8660e902-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.13.238.189@KRBTEST.COM'} at 127.0.0.1:33524
I20260929 03:15:44.826064 20399 tablet_copy_service.cc:433] P b4bc38a6af4a493dbba8a0c697b98c77: ending tablet copy session 12996e1d57454853ba637b4f8660e902-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 12996e1d57454853ba637b4f8660e902
I20260929 03:15:44.832947 20843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:44.834502 20956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:44.834604 20953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:44.834794 20954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:44.834822 20843 server_base.cc:1034] running on GCE node
I20260929 03:15:44.835078 20843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:44.835343 20843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:44.836479 20843 hybrid_clock.cc:648] HybridClock initialized: now 1790651744836471 us; error 24 us; skew 500 ppm
Sep 29 03:15:44 dist-test-slave-nbjg krb5kdc[20334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651744, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:44.837808 20843 init.cc:377] Logged in from keytab as oryx/127.13.238.189@KRBTEST.COM (short username oryx)
I20260929 03:15:44.838255 20843 webserver.cc:460] Webserver started at http://127.13.238.189:33447/ using document root <none> and password file <none>
I20260929 03:15:44.838431 20843 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:44.838465 20843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:44.839488 20843 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:15:44.840289 20963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:44.840502 20843 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:15:44.840570 20843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-1/wal
uuid: "12996e1d57454853ba637b4f8660e902"
format_stamp: "Formatted at 2026-09-29 03:15:44 on dist-test-slave-nbjg"
server_key: "1d957f90c0af847ce4e51d004764a14c"
server_key_iv: "1c70137ea9ddb8aa44d0094ff0cb6685"
server_key_version: "encryptionkey@0"
I20260929 03:15:44.840648 20843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:44.848026 20843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:44.849722 20843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:44.854086 20843 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.189:46533
I20260929 03:15:44.854423 21015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.189:46533 every 8 connection(s)
I20260929 03:15:44.854691 20843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-1/data/info.pb
I20260929 03:15:44.856381 21016 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:15:44.857445 21016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902: Bootstrap starting.
I20260929 03:15:44.868284 21016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:15:44.868568 21016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902: Bootstrap complete.
I20260929 03:15:44.869060 21016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: NON_VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: true } }
I20260929 03:15:44.869174 21016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 12996e1d57454853ba637b4f8660e902, State: Initialized, Role: LEARNER
I20260929 03:15:44.869273 21016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: NON_VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: true } }
I20260929 03:15:44.869591 21019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: NON_VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: true } } }
I20260929 03:15:44.869730 21016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:44.869750 21019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [sys.catalog]: This master's current role is: LEARNER
I20260929 03:15:44.872313 21030 catalog_manager.cc:1259] Loaded cluster ID: e30d83aca1b1425e8a563e680cc2740b
I20260929 03:15:44.872440 21030 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902: loading cluster ID for follower catalog manager: success
I20260929 03:15:44.874125 21030 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902: acquiring CA information for follower catalog manager: success
I20260929 03:15:44.875353 21030 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 03:15:44.951844 20952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:44.952121 20952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:44.952180 20952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:44.956382 20952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:44.956452 20952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:44.956480 20952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:44.956504 20952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:44.956570 20952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:15:44.963008 20952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:43675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:43921,127.13.238.189:46533,127.13.238.188:44881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.188.keytab
--principal=oryx/127.13.238.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.188:44881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.238.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:44.964139 20952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:44.965178 20952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:44.973381 21038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:44.973388 21035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:44.973441 20952 server_base.cc:1034] running on GCE node
W20260929 03:15:44.973392 21036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:44.974146 20952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:44.975225 20952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:44.976418 20952 hybrid_clock.cc:648] HybridClock initialized: now 1790651744976393 us; error 38 us; skew 500 ppm
Sep 29 03:15:44 dist-test-slave-nbjg krb5kdc[20334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651744, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:44.979380 20952 init.cc:377] Logged in from keytab as oryx/127.13.238.188@KRBTEST.COM (short username oryx)
I20260929 03:15:44.980249 20952 webserver.cc:460] Webserver started at http://127.13.238.188:33755/ using document root <none> and password file <none>
I20260929 03:15:44.980862 20952 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:44.980940 20952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:44.981151 20952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:44.983261 20952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-2/data/instance:
uuid: "02939efbf00847fa8d405999eb01407e"
format_stamp: "Formatted at 2026-09-29 03:15:44 on dist-test-slave-nbjg"
server_key: "d809d14949aa1b611b6555b0bb4181fd"
server_key_iv: "f1c8e9a8d55e83c0c416c07511cc3d25"
server_key_version: "encryptionkey@0"
I20260929 03:15:44.983788 20952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-2/wal/instance:
uuid: "02939efbf00847fa8d405999eb01407e"
format_stamp: "Formatted at 2026-09-29 03:15:44 on dist-test-slave-nbjg"
server_key: "d809d14949aa1b611b6555b0bb4181fd"
server_key_iv: "f1c8e9a8d55e83c0c416c07511cc3d25"
server_key_version: "encryptionkey@0"
I20260929 03:15:44.987576 20952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:15:44.990048 21045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:44.991307 20952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:44.991420 20952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-2/wal
uuid: "02939efbf00847fa8d405999eb01407e"
format_stamp: "Formatted at 2026-09-29 03:15:44 on dist-test-slave-nbjg"
server_key: "d809d14949aa1b611b6555b0bb4181fd"
server_key_iv: "f1c8e9a8d55e83c0c416c07511cc3d25"
server_key_version: "encryptionkey@0"
I20260929 03:15:44.991511 20952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:45.000312 20952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:45.003417 20952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:45.003582 20952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:45.010052 20952 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.188:44881
I20260929 03:15:45.010339 21097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.188:44881 every 8 connection(s)
I20260929 03:15:45.011059 20952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-2/data/info.pb
I20260929 03:15:45.013854 21098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:45.016706 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 20952
I20260929 03:15:45.018311 21098 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:15:45 dist-test-slave-nbjg krb5kdc[20334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651744, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.188@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM
Sep 29 03:15:45 dist-test-slave-nbjg krb5kdc[20334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.238.188@KRBTEST.COM
I20260929 03:15:45.031814 21100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.018441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56450 (local address 127.13.238.188:44881)
0929 03:15:45.018942 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.018958 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.018998 (+ 40us) server_negotiation.cc:407] Connection header received
0929 03:15:45.019664 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.019682 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.020141 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.020572 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:45.021419 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.022858 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.024123 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.024627 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.024824 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.027726 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:45.027732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:45.027744 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:45.027764 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:45.029103 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.029501 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.029506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.029511 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.029610 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.029857 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.029860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.029862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.030290 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:45.030464 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.030756 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.031031 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":150,"threads_started":1}
I20260929 03:15:45.033550 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.020317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33528 (local address 127.13.238.190:43921)
0929 03:15:45.020433 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.020437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.020836 (+ 399us) server_negotiation.cc:407] Connection header received
0929 03:15:45.021082 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.021085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.021133 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.021200 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:45.022009 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.022859 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.024301 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.024624 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.024738 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.027396 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:45.027400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:45.027402 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:45.027429 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:45.031616 (+ 4187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.032325 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.032329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.032330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.032412 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.032660 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.032662 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.032664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.032862 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:45.032956 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.033329 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.033433 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:15:45.033607 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.020195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.0.0.1:33528)
0929 03:15:45.020689 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:45.020747 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:45.020952 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:45.021286 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:45.021292 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:45.021650 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:45.021815 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.021826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.022989 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.022993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.024176 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.024186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.024780 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.024784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.024913 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.025488 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:45.025509 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:45.027228 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:45.031764 (+ 4536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.031772 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.031789 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.032218 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.032496 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.032499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.032501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.032578 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.033005 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:45.033011 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:45.033119 (+ 108us) client_negotiation.cc:769] Sending connection context
0929 03:15:45.033257 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 03:15:45.033439 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":133,"threads_started":1}
I20260929 03:15:45.034790 21098 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:15:45 dist-test-slave-nbjg krb5kdc[20334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651744, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.188@KRBTEST.COM for oryx/127.13.238.189@KRBTEST.COM
I20260929 03:15:45.041939 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.035207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:46533 (local address 127.0.0.1:34494)
0929 03:15:45.035345 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:45.035359 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:45.035436 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:45.035799 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:45.035801 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:45.035923 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:45.036014 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.036019 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.036845 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.036848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.037398 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.037406 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.037783 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.037786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.037856 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.038220 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:45.038234 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:45.039298 (+ 1064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:45.040791 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.040794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.040795 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.040964 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.041181 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.041184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.041185 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.041263 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.041547 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:45.041549 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:45.041613 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 03:15:45.041684 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 03:15:45.041824 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 03:15:45.041957 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.035346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34494 (local address 127.13.238.189:46533)
0929 03:15:45.035608 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.035611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.035623 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:15:45.035658 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.035660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.035698 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.035769 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:45.036096 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.036754 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.037482 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.037703 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.037771 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.039438 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:45.039441 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:45.039443 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:45.039465 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:45.040691 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.041038 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.041041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.041042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.041115 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.041325 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.041327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.041328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.041485 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:45.041553 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.041722 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.041832 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":116,"threads_started":1}
I20260929 03:15:45.042650 21098 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.188" port: 44881 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:15:45 dist-test-slave-nbjg krb5kdc[20334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651744, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.188@KRBTEST.COM for oryx/127.13.238.188@KRBTEST.COM
I20260929 03:15:45.049129 21100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.043116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56452 (local address 127.13.238.188:44881)
0929 03:15:45.043204 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.043208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.043220 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:15:45.043318 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.043320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.043357 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.043415 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:45.043710 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.044224 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.044901 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.045137 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.045218 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.046811 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:45.046815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:45.046818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:45.046837 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:45.047847 (+ 1010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.048178 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.048181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.048183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.048247 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.048458 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.048461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.048463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.048609 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:45.048687 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.048893 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.049007 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":38}
I20260929 03:15:45.049129 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.043068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:44881 (local address 127.0.0.1:56452)
0929 03:15:45.043156 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:45.043169 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:45.043236 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:45.043433 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:45.043436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:45.043565 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:45.043621 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.043626 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.044327 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.044331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.044810 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.044817 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.045222 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.045225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.045304 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.045622 (+ 318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:45.045632 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:45.046699 (+ 1067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:45.047938 (+ 1239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.047940 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.047942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.048102 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.048321 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.048324 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.048326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.048389 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.048687 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:45.048690 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:45.048765 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 03:15:45.048844 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 03:15:45.048963 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 03:15:45.051992 20570 heartbeater.cc:503] Master 127.13.238.190:43921 was elected leader, sending a full tablet report...
I20260929 03:15:45.052438 21098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e: Bootstrap starting.
I20260929 03:15:45.054885 21098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:45.055578 21098 log.cc:824] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:45.057502 21098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e: No bootstrap required, opened a new log
I20260929 03:15:45.060433 21098 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 44881 } }
I20260929 03:15:45.060632 21098 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:45.060686 21098 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02939efbf00847fa8d405999eb01407e, State: Initialized, Role: FOLLOWER
I20260929 03:15:45.061125 21098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 44881 } }
I20260929 03:15:45.061559 21104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 44881 } } }
I20260929 03:15:45.061674 21104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:15:45.062319 21098 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [sys.catalog]: configured and running, proceeding with master startup.
W20260929 03:15:45.066939 21115 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:15:45.067014 21115 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:15:45.070264 20952 master_runner.cc:428] Detected that this master 02939efbf00847fa8d405999eb01407e is joining an existing cluster
I20260929 03:15:45.070339 20952 master_runner.cc:432] Initiating AddMaster RPC to add 127.13.238.188:44881
I20260929 03:15:45.085484 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 20441
I20260929 03:15:45.091780 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:39585
--local_ip_for_outbound_sockets=127.13.238.129
--tserver_master_addrs=127.13.238.190:43921
--webserver_port=46285
--webserver_interface=127.13.238.129
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.129.keytab
--principal=oryx/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:43675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.238.190:43921,127.13.238.189:46533
--tserver_master_addrs=127.13.238.190:43921,127.13.238.189:46533,127.13.238.188:44881 with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:15:45.093560 21100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.072753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56454 (local address 127.13.238.188:44881)
0929 03:15:45.072863 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.072867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.073658 (+ 791us) server_negotiation.cc:407] Connection header received
0929 03:15:45.073743 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.073746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.073788 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.073859 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:45.074471 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.075204 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.076335 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.076668 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.076735 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.078130 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:45.078133 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:45.078136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:45.078158 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:45.091962 (+ 13804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.092388 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.092391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.092393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.092455 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.092667 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.092670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.092672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.092871 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:45.092931 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.093248 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.093377 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:15:45.094264 21121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.072670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:44881 (local address 127.0.0.1:56454)
0929 03:15:45.073561 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:45.073573 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:45.073675 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:45.073954 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:45.073957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:45.074233 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:45.074358 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.074365 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.075332 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.075335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.076209 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.076219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.076875 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.076878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.076963 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.077499 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:45.077514 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:45.077987 (+ 473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:45.092072 (+ 14085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.092075 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.092076 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.092301 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.092533 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.092535 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.092537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.092598 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.092983 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:45.092986 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:45.093069 (+ 83us) client_negotiation.cc:769] Sending connection context
0929 03:15:45.093836 (+ 767us) client_negotiation.cc:240] Negotiation successful
0929 03:15:45.094037 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":72,"threads_started":1}
I20260929 03:15:45.096756 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.073195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34508 (local address 127.13.238.189:46533)
0929 03:15:45.073319 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.073324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.073929 (+ 605us) server_negotiation.cc:407] Connection header received
0929 03:15:45.074087 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.074090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.074137 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.074185 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:45.074745 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.075449 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.076630 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.076959 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.077026 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.091923 (+ 14897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:45.091927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:45.091931 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:45.091963 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:45.093546 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.094430 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.094434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.094436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.094520 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.095850 (+ 1330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.095854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.095856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.096070 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:45.096147 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.096481 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.096608 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:15:45.097257 21122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.073066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:46533 (local address 127.0.0.1:34508)
0929 03:15:45.073842 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:45.073854 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:45.073987 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:45.074348 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:45.074351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:45.074519 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:45.074621 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.074628 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.075575 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.075578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.076506 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.076517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.077188 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.077191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.077271 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.090924 (+ 13653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:45.090942 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:45.091676 (+ 734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:45.093672 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.093676 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.093678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.094133 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.094625 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.094628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.094630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.094708 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.096201 (+ 1493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:45.096204 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:45.096302 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 03:15:45.096950 (+ 648us) client_negotiation.cc:240] Negotiation successful
0929 03:15:45.097092 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":23,"thread_start_us":77,"threads_started":1}
I20260929 03:15:45.098321 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.073363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33530 (local address 127.13.238.190:43921)
0929 03:15:45.073471 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.073475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.073489 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:15:45.073530 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.073533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.073583 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.073656 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:45.074325 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.075122 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.076824 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.077651 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.077711 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.094856 (+ 17145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:45.094860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:45.094863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:45.094895 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:45.096538 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.096997 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.097001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.097003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.097081 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.097363 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.097366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.097368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.097572 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:45.097654 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.097992 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.098131 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 03:15:45.098780 21123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.072609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.0.0.1:33530)
0929 03:15:45.073314 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:45.073330 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:45.073421 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:45.073776 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:45.073778 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:45.074050 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:45.074176 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.074183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.075252 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.075255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.076715 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.076723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.077835 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.077838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.077906 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.078344 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:45.078358 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:45.094068 (+ 15710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:45.096662 (+ 2594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.096665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.096668 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.096885 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.097180 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.097183 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.097185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.097270 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.097723 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:45.097726 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:45.097812 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 03:15:45.098474 (+ 662us) client_negotiation.cc:240] Negotiation successful
0929 03:15:45.098615 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":3,"thread_start_us":140,"threads_started":1}
Sep 29 03:15:45 dist-test-slave-nbjg krb5kdc[20334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651743, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.190@KRBTEST.COM for oryx/127.13.238.188@KRBTEST.COM
I20260929 03:15:45.111917 21100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.103325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56470 (local address 127.13.238.188:44881)
0929 03:15:45.103444 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.103448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.103485 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:15:45.103647 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.103651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.103706 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.103789 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:45.104200 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.104995 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.106094 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.106457 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.106552 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.108728 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:45.108732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:45.108735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:45.108759 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:45.110339 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.110790 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.110793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.110795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.110873 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.111123 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.111126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.111128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.111334 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:45.111422 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.111658 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.111764 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:15:45.111918 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.103266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:44881 (local address 127.0.0.1:56470)
0929 03:15:45.103431 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:45.103444 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:45.103522 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:45.103811 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:45.103814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:45.103990 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:45.104092 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.104098 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.105094 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.105098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.105959 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.105969 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.106556 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.106560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.106661 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.107083 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:45.107096 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:45.108603 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:45.110454 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.110458 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.110460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.110702 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.110955 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.110959 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.110961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.111049 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.111415 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:45.111418 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:45.111513 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 03:15:45.111608 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 03:15:45.111746 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 03:15:45.113191 20379 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.13.238.188:44881: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "02939efbf00847fa8d405999eb01407e"
member_type: NON_VOTER
last_known_addr {
host: "127.13.238.188"
port: 44881
}
attrs {
promote: true
}
}
dest_uuid: "b4bc38a6af4a493dbba8a0c697b98c77"
cas_config_opid_index: 5
I20260929 03:15:45.113829 20379 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: NON_VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: true } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: NON_VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: true } }
I20260929 03:15:45.114913 20940 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 02939efbf00847fa8d405999eb01407e (127.13.238.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: NON_VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: true } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: NON_VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: true } } }
I20260929 03:15:45.116297 20940 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.13.238.188:44881
I20260929 03:15:45.116477 20940 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: NON_VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: true } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: NON_VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: true } } }
I20260929 03:15:45.116573 20940 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:45.117430 21072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:45.118794 21072 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [term 1 FOLLOWER]: Refusing update from remote peer b4bc38a6af4a493dbba8a0c697b98c77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 03:15:45.119454 20939 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: NON_VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:45.120936 20952 master.cc:537] Master@127.13.238.188:44881 shutting down...
I20260929 03:15:45.123950 21072 raft_consensus.cc:858] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 44881 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: NON_VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: NON_VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: true } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: NON_VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: true } }
W20260929 03:15:45.124063 21072 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 44881 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: NON_VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: NON_VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: true } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: NON_VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: true } }
I20260929 03:15:45.125510 21072 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 12996e1d57454853ba637b4f8660e902 (127.13.238.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: NON_VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: true } } }
I20260929 03:15:45.125800 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.115272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34524 (local address 127.13.238.189:46533)
0929 03:15:45.120429 (+ 5157us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.120434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.120448 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:15:45.120497 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.120500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.120559 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.120651 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:45.121473 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.122367 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.124300 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.124961 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.125056 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.125195 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.125580 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.125666 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5057}
I20260929 03:15:45.126441 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.115038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:46533 (local address 127.0.0.1:34524)
0929 03:15:45.115471 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:45.115485 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:45.115559 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:45.120700 (+ 5141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:45.120704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:45.120725 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:45.120820 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.120826 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.122548 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.122551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.123973 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.123985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.125304 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.125308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.125407 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.125463 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 03:15:45.126174 (+ 711us) client_negotiation.cc:240] Negotiation successful
0929 03:15:45.126310 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358}
I20260929 03:15:45.127100 20990 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [term 1 LEARNER]: Refusing update from remote peer b4bc38a6af4a493dbba8a0c697b98c77: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 03:15:45.127553 20939 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: NON_VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 03:15:45.129438 20990 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 02939efbf00847fa8d405999eb01407e (127.13.238.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: NON_VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: true } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: NON_VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: true } } }
I20260929 03:15:45.130867 21104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4bc38a6af4a493dbba8a0c697b98c77. Latest consensus state: current_term: 1 leader_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: NON_VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: true } } }
I20260929 03:15:45.130970 21104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 03:15:45.131855 21019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4bc38a6af4a493dbba8a0c697b98c77. Latest consensus state: current_term: 1 leader_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: NON_VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: true } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: NON_VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: true } } }
I20260929 03:15:45.132012 21019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [sys.catalog]: This master's current role is: LEARNER
I20260929 03:15:45.132025 21104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: NON_VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: true } } }
I20260929 03:15:45.132122 21104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 03:15:45.132093 21019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: NON_VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: true } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: NON_VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: true } } }
I20260929 03:15:45.132154 21019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [sys.catalog]: This master's current role is: LEARNER
I20260929 03:15:45.132187 21104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: NON_VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: true } } }
I20260929 03:15:45.132236 21104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 03:15:45.134918 21125 mvcc.cc:205] Tried to move back new op lower bound from 7334509542514143232 to 7334509542423375872. Current Snapshot: MvccSnapshot[applied={T|T < 7334509542456553472}]
I20260929 03:15:45.136498 20939 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77: attempting to promote NON_VOTER 12996e1d57454853ba637b4f8660e902 to VOTER
I20260929 03:15:45.136948 20939 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: NON_VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: true } }
I20260929 03:15:45.138268 20990 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [term 1 LEARNER]: Refusing update from remote peer b4bc38a6af4a493dbba8a0c697b98c77: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 03:15:45.138690 20940 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260929 03:15:45.139853 20365 consensus_peers.cc:487] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 -> Peer 02939efbf00847fa8d405999eb01407e (127.13.238.188:44881): Couldn't send request to peer 02939efbf00847fa8d405999eb01407e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 03:15:45.139950 20952 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260929 03:15:45.140201 20952 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260929 03:15:45.140296 20952 tablet_replica.cc:330] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e: stopping tablet replica
I20260929 03:15:45.140352 20940 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 12996e1d57454853ba637b4f8660e902 (127.13.238.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: NON_VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: true } } }
I20260929 03:15:45.141779 20940 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: NON_VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: true } } }
I20260929 03:15:45.141881 20940 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:45.142529 20939 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: NON_VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: true } } }
I20260929 03:15:45.142630 20939 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:45.143199 20990 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 12996e1d57454853ba637b4f8660e902 (127.13.238.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: NON_VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: true } } }
I20260929 03:15:45.144615 21130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: NON_VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: true } } }
I20260929 03:15:45.144917 21130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:15:45.146409 20952 master.cc:552] Master@127.13.238.188:44881 shutdown complete.
I20260929 03:15:45.146456 20952 master_runner.cc:305] Clearing existing system tablet
I20260929 03:15:45.146672 20952 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:45.146720 20952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:45.147912 20952 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:45.148722 21137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:45.148938 20952 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:45.149115 20952 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:45.149175 20952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-2/wal
uuid: "02939efbf00847fa8d405999eb01407e"
format_stamp: "Formatted at 2026-09-29 03:15:44 on dist-test-slave-nbjg"
server_key: "d809d14949aa1b611b6555b0bb4181fd"
server_key_iv: "f1c8e9a8d55e83c0c416c07511cc3d25"
server_key_version: "encryptionkey@0"
I20260929 03:15:45.151031 20952 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:45.153478 20952 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 03:15:45.153609 20952 log.cc:1196] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 03:15:45.153898 20952 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e: Deleting consensus metadata
I20260929 03:15:45.154071 20952 master_runner.cc:315] Copying system tablet from 127.13.238.190:43921
I20260929 03:15:45.155270 20952 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e: tablet copy: Beginning tablet copy session from remote peer at address 127.13.238.190:43921
I20260929 03:15:45.166914 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.158597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33542 (local address 127.13.238.190:43921)
0929 03:15:45.158726 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.158731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.158967 (+ 236us) server_negotiation.cc:407] Connection header received
0929 03:15:45.159094 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.159098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.159155 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.159228 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:45.159790 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.160625 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.161769 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.162083 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.162178 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.163588 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:45.163592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:45.163595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:45.163622 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:45.165162 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.165628 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.165632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.165634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.165718 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.165999 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.166001 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.166003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.166229 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:45.166297 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.166640 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.166754 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:15:45.167461 21143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.158507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.0.0.1:33542)
0929 03:15:45.158859 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:45.158874 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:45.158993 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:45.159264 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:45.159267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:45.159526 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:45.159640 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.159647 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.160725 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.160729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.161643 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.161653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.162355 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.162360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.162456 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.163000 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:45.163015 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:45.163438 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:45.165278 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.165282 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.165284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.165525 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.165818 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.165821 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.165823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.165906 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.166373 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:45.166376 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:45.166470 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 03:15:45.167145 (+ 675us) client_negotiation.cc:240] Negotiation successful
0929 03:15:45.167291 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":88,"threads_started":1}
I20260929 03:15:45.168021 20399 tablet_copy_service.cc:140] P b4bc38a6af4a493dbba8a0c697b98c77: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 02939efbf00847fa8d405999eb01407e ({username='oryx', principal='oryx/127.13.238.188@KRBTEST.COM'} at 127.0.0.1:33542)
I20260929 03:15:45.168119 20399 tablet_copy_service.cc:161] P b4bc38a6af4a493dbba8a0c697b98c77: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 02939efbf00847fa8d405999eb01407e at {username='oryx', principal='oryx/127.13.238.188@KRBTEST.COM'} at 127.0.0.1:33542: session id = 02939efbf00847fa8d405999eb01407e-00000000000000000000000000000000
I20260929 03:15:45.169157 20399 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 03:15:45.170851 20952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 03:15:45.173137 20952 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e: tablet copy: Starting download of 0 data blocks...
I20260929 03:15:45.173300 20952 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e: tablet copy: Starting download of 1 WAL segments...
I20260929 03:15:45.174764 20952 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 03:15:45.176473 20399 tablet_copy_service.cc:342] P b4bc38a6af4a493dbba8a0c697b98c77: Request end of tablet copy session 02939efbf00847fa8d405999eb01407e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.13.238.188@KRBTEST.COM'} at 127.0.0.1:33542
I20260929 03:15:45.176553 20399 tablet_copy_service.cc:433] P b4bc38a6af4a493dbba8a0c697b98c77: ending tablet copy session 02939efbf00847fa8d405999eb01407e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 02939efbf00847fa8d405999eb01407e
I20260929 03:15:45.183552 20952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:45.185118 21144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:45.185179 21147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:45.185585 20952 server_base.cc:1034] running on GCE node
W20260929 03:15:45.185683 21145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:45.185889 20952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:45.186162 20952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:45.187362 20952 hybrid_clock.cc:648] HybridClock initialized: now 1790651745187355 us; error 33 us; skew 500 ppm
Sep 29 03:15:45 dist-test-slave-nbjg krb5kdc[20334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651745, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:45.188961 20952 init.cc:377] Logged in from keytab as oryx/127.13.238.188@KRBTEST.COM (short username oryx)
I20260929 03:15:45.189410 20952 webserver.cc:460] Webserver started at http://127.13.238.188:37143/ using document root <none> and password file <none>
I20260929 03:15:45.189579 20952 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:45.189608 20952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:45.191099 20952 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:45.192129 21154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:45.192384 20952 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:45.192484 20952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-2/wal
uuid: "02939efbf00847fa8d405999eb01407e"
format_stamp: "Formatted at 2026-09-29 03:15:44 on dist-test-slave-nbjg"
server_key: "d809d14949aa1b611b6555b0bb4181fd"
server_key_iv: "f1c8e9a8d55e83c0c416c07511cc3d25"
server_key_version: "encryptionkey@0"
I20260929 03:15:45.192618 20952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:45.217118 20952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:45.219393 20952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:45.224180 20952 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.188:44881
I20260929 03:15:45.224624 21206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.188:44881 every 8 connection(s)
I20260929 03:15:45.225497 20952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/master-2/data/info.pb
I20260929 03:15:45.228389 21207 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:15:45.229372 21207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e: Bootstrap starting.
I20260929 03:15:45.236328 21207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:15:45.236606 21207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e: Bootstrap complete.
I20260929 03:15:45.237176 21207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: NON_VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: true } }
I20260929 03:15:45.237294 21207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 02939efbf00847fa8d405999eb01407e, State: Initialized, Role: LEARNER
I20260929 03:15:45.237398 21207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: NON_VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: true } }
I20260929 03:15:45.237707 21213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: NON_VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: true } } }
I20260929 03:15:45.237800 21207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:45.237833 21213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [sys.catalog]: This master's current role is: LEARNER
I20260929 03:15:45.241293 21224 catalog_manager.cc:1259] Loaded cluster ID: e30d83aca1b1425e8a563e680cc2740b
I20260929 03:15:45.241362 21224 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e: loading cluster ID for follower catalog manager: success
W20260929 03:15:45.243075 21124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:45.243432 21124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:45.243507 21124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 03:15:45.243506 21224 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e: acquiring CA information for follower catalog manager: success
I20260929 03:15:45.244879 21224 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 03:15:45.248178 21124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:45.248229 21124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:45.248288 21124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:15:45.252892 20704 heartbeater.cc:503] Master 127.13.238.190:43921 was elected leader, sending a full tablet report...
I20260929 03:15:45.253680 21124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:43675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.129.keytab
--principal=oryx/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:39585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=46285
--tserver_master_addrs=127.13.238.190:43921,127.13.238.189:46533,127.13.238.188:44881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:45.254858 21124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:45.255864 21124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:45.265376 21229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:45.265377 21226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:45.265377 21227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:45.265614 21124 server_base.cc:1034] running on GCE node
I20260929 03:15:45.266005 21124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:45.266619 21124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:45.267765 21124 hybrid_clock.cc:648] HybridClock initialized: now 1790651745267740 us; error 37 us; skew 500 ppm
Sep 29 03:15:45 dist-test-slave-nbjg krb5kdc[20334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651745, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:45.270398 21124 init.cc:377] Logged in from keytab as oryx/127.13.238.129@KRBTEST.COM (short username oryx)
I20260929 03:15:45.271281 21124 webserver.cc:460] Webserver started at http://127.13.238.129:46285/ using document root <none> and password file <none>
I20260929 03:15:45.271811 21124 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:45.271890 21124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:45.275681 21124 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 03:15:45.277742 21236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:45.279007 21124 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:15:45.279114 21124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "4350ad5a78764d1db39c247d759ce213"
format_stamp: "Formatted at 2026-09-29 03:15:43 on dist-test-slave-nbjg"
server_key: "b0c40ca6edd75fdba3d3c69c12e00dcf"
server_key_iv: "0e732300ba6d63961a2819f9c13b586a"
server_key_version: "encryptionkey@0"
I20260929 03:15:45.279641 21124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:45.295815 21124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:45.298943 21124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:45.299118 21124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:45.299703 21124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:45.300607 21124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:45.300693 21124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:45.300756 21124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:45.300781 21124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:45.308913 21124 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:39585
I20260929 03:15:45.308975 21349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:39585 every 8 connection(s)
I20260929 03:15:45.309873 21124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:15:45.312085 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 21124
I20260929 03:15:45.312273 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 20575
I20260929 03:15:45.320514 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:35993
--local_ip_for_outbound_sockets=127.13.238.130
--tserver_master_addrs=127.13.238.190:43921
--webserver_port=35891
--webserver_interface=127.13.238.130
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.130.keytab
--principal=oryx/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:43675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.238.190:43921,127.13.238.189:46533
--tserver_master_addrs=127.13.238.190:43921,127.13.238.189:46533,127.13.238.188:44881 with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:15:45 dist-test-slave-nbjg krb5kdc[20334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651745, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.188@KRBTEST.COM
Sep 29 03:15:45 dist-test-slave-nbjg krb5kdc[20334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651745, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.189@KRBTEST.COM
Sep 29 03:15:45 dist-test-slave-nbjg krb5kdc[20334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651745, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM
I20260929 03:15:45.326921 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.313347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:41219 (local address 127.13.238.189:46533)
0929 03:15:45.313475 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.313479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.314075 (+ 596us) server_negotiation.cc:407] Connection header received
0929 03:15:45.316096 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.316100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.316151 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.316198 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:45.316716 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.317481 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.318726 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.319038 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.319110 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.322968 (+ 3858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:45.322972 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:45.322975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:45.323003 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:45.324504 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.325181 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.325185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.325187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.325273 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.325673 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.325677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.325678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.325890 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:45.325963 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.326639 (+ 676us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.326759 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 03:15:45.326965 21354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.311729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:54787 (local address 127.13.238.188:44881)
0929 03:15:45.311982 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.311986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.313858 (+ 1872us) server_negotiation.cc:407] Connection header received
0929 03:15:45.314615 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.314619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.314673 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.314727 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:45.316020 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.317029 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.318707 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.319062 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.319125 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.322978 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:45.322982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:45.322985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:45.323011 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:45.324660 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.325209 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.325213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.325214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.325317 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.325674 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.325677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.325679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.325899 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:45.325982 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.326664 (+ 682us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.326793 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":78,"threads_started":1}
I20260929 03:15:45.327536 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.311675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:35069 (local address 127.13.238.190:43921)
0929 03:15:45.311833 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.311838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.313544 (+ 1706us) server_negotiation.cc:407] Connection header received
0929 03:15:45.314593 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.314596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.314655 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.314725 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:45.315999 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.316810 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.318707 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.319052 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.319125 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.324295 (+ 5170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:45.324299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:45.324302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:45.324332 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:45.325859 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.326322 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.326325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.326327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.326410 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.326693 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.326696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.326697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.326901 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:45.326977 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.327286 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.327396 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 03:15:45.328466 21357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.312185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.13.238.129:35069)
0929 03:15:45.313454 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:45.313467 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:45.314392 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:45.314890 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:45.314897 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:45.315327 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:45.315860 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.315873 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.316973 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.316976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.318575 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.318586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.319310 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.319314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.319631 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.320463 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:45.320477 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:45.324141 (+ 3664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:45.325979 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.325982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.325984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.326216 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.326515 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.326518 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.326520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.326602 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.327034 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:45.327037 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:45.327117 (+ 80us) client_negotiation.cc:769] Sending connection context
0929 03:15:45.327724 (+ 607us) client_negotiation.cc:240] Negotiation successful
0929 03:15:45.327914 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1180,"mutex_wait_us":364,"spinlock_wait_cycles":92800,"thread_start_us":135,"threads_started":1}
I20260929 03:15:45.328467 21356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.312185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:44881 (local address 127.13.238.129:54787)
0929 03:15:45.313158 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:45.313211 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:45.314394 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:45.314892 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:45.314898 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:45.315330 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:45.315862 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.315875 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.317142 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.317145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.318587 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.318594 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.319310 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.319314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.319615 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.320357 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:45.320372 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:45.322774 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:45.324786 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.324789 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.324791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.325100 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.325428 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.325431 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.325433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.325568 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.326042 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:45.326045 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:45.326372 (+ 327us) client_negotiation.cc:769] Sending connection context
0929 03:15:45.327515 (+ 1143us) client_negotiation.cc:240] Negotiation successful
0929 03:15:45.327845 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":257,"spinlock_wait_cycles":13952,"thread_start_us":140,"threads_started":1}
I20260929 03:15:45.328467 21355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.313229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:46533 (local address 127.13.238.129:41219)
0929 03:15:45.313986 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:45.313998 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:45.315008 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:45.316330 (+ 1322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:45.316333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:45.316494 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:45.316590 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.316596 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.317610 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.317613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.318599 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.318610 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.319310 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.319313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.319593 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.320229 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:45.320250 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:45.322772 (+ 2522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:45.324620 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.324626 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.324643 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.325071 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.325395 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.325397 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.325399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.325564 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.326018 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:45.326024 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:45.326371 (+ 347us) client_negotiation.cc:769] Sending connection context
0929 03:15:45.327127 (+ 756us) client_negotiation.cc:240] Negotiation successful
0929 03:15:45.327453 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":105,"spinlock_wait_cycles":8064,"thread_start_us":67,"threads_started":1}
I20260929 03:15:45.329841 21350 heartbeater.cc:347] Connected to a master server at 127.13.238.188:44881
I20260929 03:15:45.329993 21352 heartbeater.cc:347] Connected to a master server at 127.13.238.189:46533
I20260929 03:15:45.330101 21352 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:45.329828 21351 heartbeater.cc:347] Connected to a master server at 127.13.238.190:43921
I20260929 03:15:45.330103 21350 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:45.330302 21351 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:45.330700 21352 heartbeater.cc:511] Master 127.13.238.189:46533 requested a full tablet report, sending...
I20260929 03:15:45.330715 21351 heartbeater.cc:511] Master 127.13.238.190:43921 requested a full tablet report, sending...
I20260929 03:15:45.330700 21350 heartbeater.cc:511] Master 127.13.238.188:44881 requested a full tablet report, sending...
I20260929 03:15:45.332046 20379 ts_manager.cc:194] Re-registered known tserver with Master: 4350ad5a78764d1db39c247d759ce213 (127.13.238.129:39585)
I20260929 03:15:45.332252 21171 ts_manager.cc:194] Registered new tserver with Master: 4350ad5a78764d1db39c247d759ce213 (127.13.238.129:39585)
I20260929 03:15:45.332252 20980 ts_manager.cc:194] Registered new tserver with Master: 4350ad5a78764d1db39c247d759ce213 (127.13.238.129:39585)
I20260929 03:15:45.332674 20379 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.238.129@KRBTEST.COM'} at 127.13.238.129:35069
W20260929 03:15:45.437026 21358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:45.437284 21358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:45.437331 21358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:45.441807 21358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:45.441879 21358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:45.441949 21358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:15:45.447938 21358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:43675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.130.keytab
--principal=oryx/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:35993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=35891
--tserver_master_addrs=127.13.238.190:43921,127.13.238.189:46533,127.13.238.188:44881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:45.449323 21358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:45.450664 21358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:45.460475 21363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:45.460472 21364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:45.460552 21358 server_base.cc:1034] running on GCE node
W20260929 03:15:45.460472 21366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:45.461124 21358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:45.461802 21358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:45.462963 21358 hybrid_clock.cc:648] HybridClock initialized: now 1790651745462934 us; error 42 us; skew 500 ppm
I20260929 03:15:45.466310 20838 heartbeater.cc:503] Master 127.13.238.190:43921 was elected leader, sending a full tablet report...
Sep 29 03:15:45 dist-test-slave-nbjg krb5kdc[20334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651745, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:45.469132 21358 init.cc:377] Logged in from keytab as oryx/127.13.238.130@KRBTEST.COM (short username oryx)
I20260929 03:15:45.470028 21358 webserver.cc:460] Webserver started at http://127.13.238.130:35891/ using document root <none> and password file <none>
I20260929 03:15:45.470647 21358 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:45.470741 21358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:45.474560 21358 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 03:15:45.476616 21373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:45.477834 21358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:15:45.477937 21358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "2ae2e4cfa8764352a502fbe7ff3009c4"
format_stamp: "Formatted at 2026-09-29 03:15:44 on dist-test-slave-nbjg"
server_key: "87147124e12f965d2fc7f42df1e1b63c"
server_key_iv: "c2b306599571989f2265c8e5b7ad94ee"
server_key_version: "encryptionkey@0"
I20260929 03:15:45.478543 21358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:45.488622 21358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:45.491573 21358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:45.491734 21358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:45.492290 21358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:45.493208 21358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:45.493263 21358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:45.493333 21358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:45.493359 21358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:45.502058 21358 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:35993
I20260929 03:15:45.502081 21486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:35993 every 8 connection(s)
I20260929 03:15:45.503096 21358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:15:45.506980 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 21358
I20260929 03:15:45.507174 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 20709
I20260929 03:15:45.513900 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:38001
--local_ip_for_outbound_sockets=127.13.238.131
--tserver_master_addrs=127.13.238.190:43921
--webserver_port=32795
--webserver_interface=127.13.238.131
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.131.keytab
--principal=oryx/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:43675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.238.190:43921,127.13.238.189:46533
--tserver_master_addrs=127.13.238.190:43921,127.13.238.189:46533,127.13.238.188:44881 with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:15:45 dist-test-slave-nbjg krb5kdc[20334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651745, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.189@KRBTEST.COM
Sep 29 03:15:45 dist-test-slave-nbjg krb5kdc[20334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651745, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.188@KRBTEST.COM
Sep 29 03:15:45 dist-test-slave-nbjg krb5kdc[20334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651745, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM
I20260929 03:15:45.519759 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.506476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:55847 (local address 127.13.238.189:46533)
0929 03:15:45.506602 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.506607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.506694 (+ 87us) server_negotiation.cc:407] Connection header received
0929 03:15:45.507322 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.507325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.507378 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.507424 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:45.509279 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.510100 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.511966 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.512295 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.512363 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.515617 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:45.515622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:45.515624 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:45.515656 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:45.517246 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.517851 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.517855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.517857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.517959 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.518412 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.518416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.518418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.518656 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:45.518731 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.519458 (+ 727us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.519585 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:15:45.521646 21354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.504999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:54599 (local address 127.13.238.188:44881)
0929 03:15:45.505139 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.505144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.505957 (+ 813us) server_negotiation.cc:407] Connection header received
0929 03:15:45.509243 (+ 3286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.509247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.509311 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.509364 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:45.509946 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.510679 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.511994 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.512354 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.512426 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.516561 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:45.516565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:45.516567 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:45.516596 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:45.519263 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.519717 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.519721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.519723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.519820 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.520414 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.520418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.520420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.520614 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:45.520675 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.521377 (+ 702us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.521495 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:15:45.521759 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.506366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:46533 (local address 127.13.238.130:55847)
0929 03:15:45.506593 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:45.506604 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:45.507068 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:45.507804 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:45.507813 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:45.508302 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:45.509113 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.509132 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.510310 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.510313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.511841 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.511851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.512554 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.512557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.512763 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.513609 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:45.513631 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:45.515416 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:45.517381 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.517387 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.517405 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.517752 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.518058 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.518062 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.518064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.518292 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.518812 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:45.518818 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:45.519151 (+ 333us) client_negotiation.cc:769] Sending connection context
0929 03:15:45.519934 (+ 783us) client_negotiation.cc:240] Negotiation successful
0929 03:15:45.520339 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":169,"thread_start_us":72,"threads_started":1}
I20260929 03:15:45.522182 21492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.505311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:44881 (local address 127.13.238.130:54599)
0929 03:15:45.505796 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:45.505849 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:45.507159 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:45.509515 (+ 2356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:45.509518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:45.509690 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:45.509811 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.509817 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.510813 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.510816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.511861 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.511872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.512580 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.512583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.512792 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.513707 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:45.513730 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:45.516448 (+ 2718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:45.519376 (+ 2928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.519379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.519381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.519619 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.520219 (+ 600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.520222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.520224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.520302 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.520742 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:45.520756 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:45.520839 (+ 83us) client_negotiation.cc:769] Sending connection context
0929 03:15:45.521850 (+ 1011us) client_negotiation.cc:240] Negotiation successful
0929 03:15:45.521991 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":209,"spinlock_wait_cycles":77952,"thread_start_us":124,"threads_started":1}
I20260929 03:15:45.522293 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.515373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:44881 (local address 127.0.0.1:56472)
0929 03:15:45.515629 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:45.515643 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:45.515728 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:45.518667 (+ 2939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:45.518671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:45.518693 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:45.518794 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.518800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.519881 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.519885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.520986 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.520995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.521885 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.521889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.521979 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.522003 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 03:15:45.522052 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 03:15:45.522123 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260929 03:15:45.522811 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.515884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56472 (local address 127.13.238.188:44881)
0929 03:15:45.518401 (+ 2517us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.518405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.518420 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:15:45.518465 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.518468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.518529 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.518901 (+ 372us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:45.519002 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.519762 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.521116 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.521752 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.522463 (+ 711us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.522603 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.522636 (+ 33us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.522700 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2430,"thread_start_us":97,"threads_started":1}
I20260929 03:15:45.523034 21489 heartbeater.cc:347] Connected to a master server at 127.13.238.189:46533
I20260929 03:15:45.523280 21489 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:45.523532 21181 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260929 03:15:45.523860 21489 heartbeater.cc:511] Master 127.13.238.189:46533 requested a full tablet report, sending...
I20260929 03:15:45.523953 21213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4bc38a6af4a493dbba8a0c697b98c77. Latest consensus state: current_term: 1 leader_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: NON_VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: true } } }
I20260929 03:15:45.524080 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.504909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:39551 (local address 127.13.238.190:43921)
0929 03:15:45.505049 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.505053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.505957 (+ 904us) server_negotiation.cc:407] Connection header received
0929 03:15:45.508069 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.508073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.508134 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.508185 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:45.509393 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.510137 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.511994 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.512354 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.512426 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.516934 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:45.516939 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:45.516941 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:45.516971 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:45.518528 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.520969 (+ 2441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.520973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.520974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.521057 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.523140 (+ 2083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.523144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.523146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.523381 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:45.523471 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.523821 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.523941 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 03:15:45.524086 21213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [sys.catalog]: This master's current role is: LEARNER
I20260929 03:15:45.524523 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.505311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.13.238.130:39551)
0929 03:15:45.505795 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:45.505849 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:45.507062 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:45.508340 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:45.508343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:45.508533 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:45.509241 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.509260 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.510312 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.510316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.511862 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.511872 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.512653 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.512656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.512773 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.513713 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:45.513731 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:45.516782 (+ 3051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:45.519897 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.519901 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.519903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.520115 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.521154 (+ 1039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.521157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.521159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.521235 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.523517 (+ 2282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:45.523520 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:45.523613 (+ 93us) client_negotiation.cc:769] Sending connection context
0929 03:15:45.524190 (+ 577us) client_negotiation.cc:240] Negotiation successful
0929 03:15:45.524351 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":701,"spinlock_wait_cycles":9600,"thread_start_us":113,"threads_started":1}
I20260929 03:15:45.524667 21134 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77: attempting to promote NON_VOTER 02939efbf00847fa8d405999eb01407e to VOTER
I20260929 03:15:45.525167 21487 heartbeater.cc:347] Connected to a master server at 127.13.238.188:44881
I20260929 03:15:45.525105 21134 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: false } }
I20260929 03:15:45.525177 21488 heartbeater.cc:347] Connected to a master server at 127.13.238.190:43921
I20260929 03:15:45.525276 21488 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:45.525278 20980 ts_manager.cc:194] Registered new tserver with Master: 2ae2e4cfa8764352a502fbe7ff3009c4 (127.13.238.130:35993)
I20260929 03:15:45.525472 21488 heartbeater.cc:511] Master 127.13.238.190:43921 requested a full tablet report, sending...
I20260929 03:15:45.525265 21487 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:45.526108 21487 heartbeater.cc:511] Master 127.13.238.188:44881 requested a full tablet report, sending...
I20260929 03:15:45.526476 20990 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [term 1 FOLLOWER]: Refusing update from remote peer b4bc38a6af4a493dbba8a0c697b98c77: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 03:15:45.526840 20379 ts_manager.cc:194] Re-registered known tserver with Master: 2ae2e4cfa8764352a502fbe7ff3009c4 (127.13.238.130:35993)
I20260929 03:15:45.527390 20379 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.238.130@KRBTEST.COM'} at 127.13.238.130:39551
I20260929 03:15:45.527921 20939 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 03:15:45.528103 21171 ts_manager.cc:194] Registered new tserver with Master: 2ae2e4cfa8764352a502fbe7ff3009c4 (127.13.238.130:35993)
I20260929 03:15:45.528640 21181 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [term 1 LEARNER]: Refusing update from remote peer b4bc38a6af4a493dbba8a0c697b98c77: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 03:15:45.529425 20939 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 03:15:45.530079 20940 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 02939efbf00847fa8d405999eb01407e (127.13.238.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: false } } }
I20260929 03:15:45.530606 20990 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 02939efbf00847fa8d405999eb01407e (127.13.238.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: false } } }
I20260929 03:15:45.531509 20940 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: false } } }
I20260929 03:15:45.531602 20940 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:45.531701 20940 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: false } } }
I20260929 03:15:45.531759 20940 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:45.531829 20940 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: false } } }
I20260929 03:15:45.531883 20940 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:45.531837 21181 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 02939efbf00847fa8d405999eb01407e (127.13.238.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: false } } }
I20260929 03:15:45.532155 21130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: false } } }
I20260929 03:15:45.532248 21130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:15:45.533214 21213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: false } } }
I20260929 03:15:45.533325 21213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [sys.catalog]: This master's current role is: FOLLOWER
W20260929 03:15:45.629311 21494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:45.629576 21494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:45.629621 21494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:45.633775 21494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:45.633848 21494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:45.633910 21494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:15:45.639374 21494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:43675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.131.keytab
--principal=oryx/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:38001
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=32795
--tserver_master_addrs=127.13.238.190:43921,127.13.238.189:46533,127.13.238.188:44881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:45.640497 21494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:45.641572 21494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:45.651398 21501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:45.651398 21504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:45.651398 21502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:45.651595 21494 server_base.cc:1034] running on GCE node
I20260929 03:15:45.652086 21494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:45.652760 21494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:45.653937 21494 hybrid_clock.cc:648] HybridClock initialized: now 1790651745653912 us; error 35 us; skew 500 ppm
Sep 29 03:15:45 dist-test-slave-nbjg krb5kdc[20334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651745, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:45.656821 21494 init.cc:377] Logged in from keytab as oryx/127.13.238.131@KRBTEST.COM (short username oryx)
I20260929 03:15:45.657815 21494 webserver.cc:460] Webserver started at http://127.13.238.131:32795/ using document root <none> and password file <none>
I20260929 03:15:45.658471 21494 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:45.658562 21494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:45.662704 21494 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:15:45.665004 21511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:45.666313 21494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:15:45.666429 21494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "bac6c1b9f519437f82e706c280a0f13f"
format_stamp: "Formatted at 2026-09-29 03:15:44 on dist-test-slave-nbjg"
server_key: "85ca053e2060f3a9afecac635d674b91"
server_key_iv: "351a3813d3929b7f1e59ffda4810ec0c"
server_key_version: "encryptionkey@0"
I20260929 03:15:45.666980 21494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:45.681897 21494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:45.684986 21494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:45.685153 21494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:45.685730 21494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:45.686612 21494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:45.686658 21494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:45.686712 21494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:45.686738 21494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:45.695210 21494 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:38001
I20260929 03:15:45.695232 21624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:38001 every 8 connection(s)
I20260929 03:15:45.696156 21494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:15:45.700788 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 21494
Sep 29 03:15:45 dist-test-slave-nbjg krb5kdc[20334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651745, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.189@KRBTEST.COM
Sep 29 03:15:45 dist-test-slave-nbjg krb5kdc[20334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651745, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.188@KRBTEST.COM
Sep 29 03:15:45 dist-test-slave-nbjg krb5kdc[20334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651745, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM
I20260929 03:15:45.713847 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.699536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:48541 (local address 127.13.238.189:46533)
0929 03:15:45.699668 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.699678 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.699782 (+ 104us) server_negotiation.cc:407] Connection header received
0929 03:15:45.700090 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.700094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.700149 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.700220 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:45.701975 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.702743 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.704464 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.704751 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.704842 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.709717 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:45.709721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:45.709723 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:45.709748 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:45.711486 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.712083 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.712086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.712088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.712164 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.712559 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.712562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.712564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.712784 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:45.712870 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.713554 (+ 684us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.713663 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:15:45.714920 21639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.703685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34526 (local address 127.13.238.189:46533)
0929 03:15:45.704615 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.704620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.704935 (+ 315us) server_negotiation.cc:407] Connection header received
0929 03:15:45.704971 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.704973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.705012 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.705083 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:45.707002 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.707773 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.708796 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.709111 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.709200 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.711196 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:45.711200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:45.711203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:45.711230 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:45.712788 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.713274 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.713278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.713279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.713506 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.713799 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.713802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.713804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.713998 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:45.714100 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.714658 (+ 558us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.714770 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":866,"thread_start_us":59,"threads_started":1}
I20260929 03:15:45.715189 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.698513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:42727 (local address 127.13.238.188:44881)
0929 03:15:45.698615 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.698619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.699349 (+ 730us) server_negotiation.cc:407] Connection header received
0929 03:15:45.700179 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.700182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.700231 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.700307 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:45.702258 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.703051 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.704492 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.704835 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.704918 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.708835 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:45.708838 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:45.708841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:45.708867 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:45.712341 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.713267 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.713270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.713272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.713377 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.713618 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.713621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.713623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.713837 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:45.714146 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.714736 (+ 590us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.714860 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:15:45.715449 21631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.699428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:46533 (local address 127.13.238.131:48541)
0929 03:15:45.699709 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:45.699721 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:45.699894 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:45.700403 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:45.700409 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:45.700947 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:45.701836 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.701856 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.702916 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.702920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.704338 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.704349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.705023 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.705027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.705305 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.706050 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:45.706063 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:45.708897 (+ 2834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:45.711604 (+ 2707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.711610 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.711630 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.711981 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.712271 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.712274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.712276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.712454 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.712949 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:45.712956 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:45.713272 (+ 316us) client_negotiation.cc:769] Sending connection context
0929 03:15:45.714006 (+ 734us) client_negotiation.cc:240] Negotiation successful
0929 03:15:45.714388 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":270,"spinlock_wait_cycles":8704,"thread_start_us":69,"threads_started":1}
I20260929 03:15:45.715968 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.698428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:44881 (local address 127.13.238.131:42727)
0929 03:15:45.699257 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:45.699269 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:45.699886 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:45.700426 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:45.700429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:45.700952 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:45.701836 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.701856 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.703174 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.703178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.704365 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.704374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.705024 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.705028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.705296 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.706141 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:45.706154 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:45.708577 (+ 2423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:45.712517 (+ 3940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.712520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.712523 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.713163 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.713452 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.713455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.713457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.713526 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.714190 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:45.714212 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:45.714298 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 03:15:45.714386 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 03:15:45.714509 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":354,"spinlock_wait_cycles":8064,"thread_start_us":640,"threads_started":1}
I20260929 03:15:45.716729 21627 heartbeater.cc:347] Connected to a master server at 127.13.238.189:46533
I20260929 03:15:45.716990 21627 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:45.717533 21627 heartbeater.cc:511] Master 127.13.238.189:46533 requested a full tablet report, sending...
I20260929 03:15:45.718120 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.697914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:43191 (local address 127.13.238.190:43921)
0929 03:15:45.698042 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.698047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.699005 (+ 958us) server_negotiation.cc:407] Connection header received
0929 03:15:45.700104 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.700107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.700162 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.700211 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:45.702255 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.703049 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.704495 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.704835 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.704907 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.709350 (+ 4443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:45.709354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:45.709357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:45.709384 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:45.711098 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.716752 (+ 5654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.716756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.716758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.716848 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.717158 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.717161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.717163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.717367 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:45.717438 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.717851 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.717971 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:15:45.718473 21625 heartbeater.cc:347] Connected to a master server at 127.13.238.188:44881
I20260929 03:15:45.718513 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.704047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33544 (local address 127.13.238.190:43921)
0929 03:15:45.705266 (+ 1219us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.705270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.705282 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:15:45.705326 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.705329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.705371 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.705450 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:45.707917 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.708723 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.709821 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.710178 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.710676 (+ 498us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.713034 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:45.713037 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:45.713040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:45.713063 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:45.716561 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.716986 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.716989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.716990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.717063 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.717355 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.717358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.717360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.717539 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:45.717673 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.718239 (+ 566us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.718370 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1147,"thread_start_us":85,"threads_started":1}
I20260929 03:15:45.718554 21625 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:45.718703 21625 heartbeater.cc:511] Master 127.13.238.188:44881 requested a full tablet report, sending...
I20260929 03:15:45.718813 20980 ts_manager.cc:194] Registered new tserver with Master: bac6c1b9f519437f82e706c280a0f13f (127.13.238.131:38001)
I20260929 03:15:45.719026 21629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.698349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.13.238.131:43191)
0929 03:15:45.698869 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:45.698909 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:45.699886 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:45.700403 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:45.700411 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:45.702028 (+ 1617us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:45.702120 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.702125 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.703184 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.703187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.704364 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.704374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.705058 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.705061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.705293 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.706011 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:45.706041 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:45.709208 (+ 3167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:45.713345 (+ 4137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.713348 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.713350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.716668 (+ 3318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.716923 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:45.716926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:45.716928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:45.717014 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:45.717492 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:45.717494 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:45.717572 (+ 78us) client_negotiation.cc:769] Sending connection context
0929 03:15:45.718716 (+ 1144us) client_negotiation.cc:240] Negotiation successful
0929 03:15:45.718865 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":208,"spinlock_wait_cycles":84992,"thread_start_us":158,"threads_started":1}
I20260929 03:15:45.719424 21171 ts_manager.cc:194] Registered new tserver with Master: bac6c1b9f519437f82e706c280a0f13f (127.13.238.131:38001)
I20260929 03:15:45.720116 21626 heartbeater.cc:347] Connected to a master server at 127.13.238.190:43921
I20260929 03:15:45.720191 21626 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:45.720341 21626 heartbeater.cc:511] Master 127.13.238.190:43921 requested a full tablet report, sending...
I20260929 03:15:45.721307 20379 ts_manager.cc:194] Re-registered known tserver with Master: bac6c1b9f519437f82e706c280a0f13f (127.13.238.131:38001)
I20260929 03:15:45.722030 20379 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.238.131@KRBTEST.COM'} at 127.13.238.131:43191
I20260929 03:15:45.723955 21354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.703856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56480 (local address 127.13.238.188:44881)
0929 03:15:45.703961 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.703965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.710465 (+ 6500us) server_negotiation.cc:407] Connection header received
0929 03:15:45.711115 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.711119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.711168 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.711218 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:45.714045 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.714762 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.717585 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.717909 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.717983 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.720958 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:45.720962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:45.720964 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:45.720987 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:45.722353 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.722784 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.722795 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.722797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.722883 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.723149 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.723153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.723154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.723352 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:45.723470 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.723687 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.723812 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:15:45.722679 20378 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:15:45.726159 20378 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:15:45.742344 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.737066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:35993 (local address 127.0.0.1:58524)
0929 03:15:45.737621 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:45.737651 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:45.737724 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:45.739060 (+ 1336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:45.739063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:45.739081 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:45.739210 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.739217 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.740144 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.740147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.740968 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.740977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.741927 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.741931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.742014 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.742035 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 03:15:45.742093 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 03:15:45.742162 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479}
I20260929 03:15:45.742619 21652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.738150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58524 (local address 127.13.238.130:35993)
0929 03:15:45.738451 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.738459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.738478 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:15:45.738550 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.738556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.738934 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.739100 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:45.739315 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.740058 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.741158 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.741825 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.741931 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.742072 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.742121 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.742481 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":127,"threads_started":1}
I20260929 03:15:45.744324 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.737342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:39585 (local address 127.0.0.1:56582)
0929 03:15:45.738638 (+ 1296us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:45.738650 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:45.738728 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:45.739495 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:45.739498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:45.739515 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:45.739603 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.739609 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.741124 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.741128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.742841 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.742851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.743936 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.743940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.744031 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.744056 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 03:15:45.744124 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 03:15:45.744201 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1236,"thread_start_us":123,"threads_started":1}
I20260929 03:15:45.744771 21650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.737514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56582 (local address 127.13.238.129:39585)
0929 03:15:45.738320 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.738334 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.738819 (+ 485us) server_negotiation.cc:407] Connection header received
0929 03:15:45.738997 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.739002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.739192 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.739732 (+ 540us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:45.739861 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.741005 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.742972 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.743815 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.743930 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.744043 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.744529 (+ 486us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.744635 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":112,"threads_started":1}
I20260929 03:15:45.745685 21421 tablet_service.cc:1467] Processing CreateTablet for tablet f2fb4a9ec2de4fd8a162536a6d90d63c (DEFAULT_TABLE table=test-table [id=56cc3ffea4c14bf0b1b14adf20e94fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:45.746752 21421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2fb4a9ec2de4fd8a162536a6d90d63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:45.747481 21284 tablet_service.cc:1467] Processing CreateTablet for tablet f2fb4a9ec2de4fd8a162536a6d90d63c (DEFAULT_TABLE table=test-table [id=56cc3ffea4c14bf0b1b14adf20e94fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:45.748347 21284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2fb4a9ec2de4fd8a162536a6d90d63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:45.754112 21653 tablet_bootstrap.cc:492] T f2fb4a9ec2de4fd8a162536a6d90d63c P 2ae2e4cfa8764352a502fbe7ff3009c4: Bootstrap starting.
I20260929 03:15:45.754684 21649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.737225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:38001 (local address 127.0.0.1:37362)
0929 03:15:45.737919 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:45.737934 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:45.738017 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:45.744851 (+ 6834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:45.744854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:45.744870 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:45.744949 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.744954 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.751779 (+ 6825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.751783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.753013 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.753036 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.754226 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.754231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.754343 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.754377 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 03:15:45.754437 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 03:15:45.754530 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":106,"threads_started":1}
I20260929 03:15:45.755720 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.738004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37362 (local address 127.13.238.131:38001)
0929 03:15:45.744254 (+ 6250us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.744264 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.744287 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:15:45.744362 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.744367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.744574 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.744704 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:45.745061 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.751631 (+ 6570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.753201 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.754043 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.755259 (+ 1216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.755422 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.755477 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.755572 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6138,"thread_start_us":101,"threads_started":1}
I20260929 03:15:45.756505 21654 tablet_bootstrap.cc:492] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213: Bootstrap starting.
I20260929 03:15:45.757249 21653 tablet_bootstrap.cc:654] T f2fb4a9ec2de4fd8a162536a6d90d63c P 2ae2e4cfa8764352a502fbe7ff3009c4: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:45.758147 21653 log.cc:824] T f2fb4a9ec2de4fd8a162536a6d90d63c P 2ae2e4cfa8764352a502fbe7ff3009c4: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:45.758360 21654 tablet_bootstrap.cc:654] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:45.759130 21654 log.cc:824] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:45.758756 21559 tablet_service.cc:1467] Processing CreateTablet for tablet f2fb4a9ec2de4fd8a162536a6d90d63c (DEFAULT_TABLE table=test-table [id=56cc3ffea4c14bf0b1b14adf20e94fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:45.759770 21559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2fb4a9ec2de4fd8a162536a6d90d63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:45.760263 21653 tablet_bootstrap.cc:492] T f2fb4a9ec2de4fd8a162536a6d90d63c P 2ae2e4cfa8764352a502fbe7ff3009c4: No bootstrap required, opened a new log
I20260929 03:15:45.760469 21653 ts_tablet_manager.cc:1397] T f2fb4a9ec2de4fd8a162536a6d90d63c P 2ae2e4cfa8764352a502fbe7ff3009c4: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 03:15:45.760785 21654 tablet_bootstrap.cc:492] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213: No bootstrap required, opened a new log
I20260929 03:15:45.760972 21654 ts_tablet_manager.cc:1397] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 03:15:45.764248 21654 raft_consensus.cc:348] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4350ad5a78764d1db39c247d759ce213" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39585 } } peers { permanent_uuid: "2ae2e4cfa8764352a502fbe7ff3009c4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35993 } } peers { permanent_uuid: "bac6c1b9f519437f82e706c280a0f13f" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38001 } }
I20260929 03:15:45.764559 21654 raft_consensus.cc:374] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:45.764614 21654 raft_consensus.cc:729] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4350ad5a78764d1db39c247d759ce213, State: Initialized, Role: FOLLOWER
I20260929 03:15:45.764782 21653 raft_consensus.cc:348] T f2fb4a9ec2de4fd8a162536a6d90d63c P 2ae2e4cfa8764352a502fbe7ff3009c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4350ad5a78764d1db39c247d759ce213" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39585 } } peers { permanent_uuid: "2ae2e4cfa8764352a502fbe7ff3009c4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35993 } } peers { permanent_uuid: "bac6c1b9f519437f82e706c280a0f13f" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38001 } }
I20260929 03:15:45.765020 21653 raft_consensus.cc:374] T f2fb4a9ec2de4fd8a162536a6d90d63c P 2ae2e4cfa8764352a502fbe7ff3009c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:45.765065 21653 raft_consensus.cc:729] T f2fb4a9ec2de4fd8a162536a6d90d63c P 2ae2e4cfa8764352a502fbe7ff3009c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae2e4cfa8764352a502fbe7ff3009c4, State: Initialized, Role: FOLLOWER
I20260929 03:15:45.765051 21654 consensus_queue.cc:260] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4350ad5a78764d1db39c247d759ce213" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39585 } } peers { permanent_uuid: "2ae2e4cfa8764352a502fbe7ff3009c4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35993 } } peers { permanent_uuid: "bac6c1b9f519437f82e706c280a0f13f" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38001 } }
I20260929 03:15:45.765516 21653 consensus_queue.cc:260] T f2fb4a9ec2de4fd8a162536a6d90d63c P 2ae2e4cfa8764352a502fbe7ff3009c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4350ad5a78764d1db39c247d759ce213" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39585 } } peers { permanent_uuid: "2ae2e4cfa8764352a502fbe7ff3009c4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35993 } } peers { permanent_uuid: "bac6c1b9f519437f82e706c280a0f13f" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38001 } }
I20260929 03:15:45.766012 21351 heartbeater.cc:503] Master 127.13.238.190:43921 was elected leader, sending a full tablet report...
I20260929 03:15:45.766047 21654 ts_tablet_manager.cc:1428] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260929 03:15:45.766866 21653 ts_tablet_manager.cc:1428] T f2fb4a9ec2de4fd8a162536a6d90d63c P 2ae2e4cfa8764352a502fbe7ff3009c4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 03:15:45.768952 21659 tablet_bootstrap.cc:492] T f2fb4a9ec2de4fd8a162536a6d90d63c P bac6c1b9f519437f82e706c280a0f13f: Bootstrap starting.
I20260929 03:15:45.769438 21488 heartbeater.cc:503] Master 127.13.238.190:43921 was elected leader, sending a full tablet report...
I20260929 03:15:45.773317 21659 tablet_bootstrap.cc:654] T f2fb4a9ec2de4fd8a162536a6d90d63c P bac6c1b9f519437f82e706c280a0f13f: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:45.774070 21659 log.cc:824] T f2fb4a9ec2de4fd8a162536a6d90d63c P bac6c1b9f519437f82e706c280a0f13f: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:45.775898 21659 tablet_bootstrap.cc:492] T f2fb4a9ec2de4fd8a162536a6d90d63c P bac6c1b9f519437f82e706c280a0f13f: No bootstrap required, opened a new log
I20260929 03:15:45.776078 21659 ts_tablet_manager.cc:1397] T f2fb4a9ec2de4fd8a162536a6d90d63c P bac6c1b9f519437f82e706c280a0f13f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 03:15:45.779632 21659 raft_consensus.cc:348] T f2fb4a9ec2de4fd8a162536a6d90d63c P bac6c1b9f519437f82e706c280a0f13f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4350ad5a78764d1db39c247d759ce213" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39585 } } peers { permanent_uuid: "2ae2e4cfa8764352a502fbe7ff3009c4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35993 } } peers { permanent_uuid: "bac6c1b9f519437f82e706c280a0f13f" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38001 } }
I20260929 03:15:45.779912 21659 raft_consensus.cc:374] T f2fb4a9ec2de4fd8a162536a6d90d63c P bac6c1b9f519437f82e706c280a0f13f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:45.779968 21659 raft_consensus.cc:729] T f2fb4a9ec2de4fd8a162536a6d90d63c P bac6c1b9f519437f82e706c280a0f13f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac6c1b9f519437f82e706c280a0f13f, State: Initialized, Role: FOLLOWER
I20260929 03:15:45.780417 21659 consensus_queue.cc:260] T f2fb4a9ec2de4fd8a162536a6d90d63c P bac6c1b9f519437f82e706c280a0f13f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4350ad5a78764d1db39c247d759ce213" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39585 } } peers { permanent_uuid: "2ae2e4cfa8764352a502fbe7ff3009c4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35993 } } peers { permanent_uuid: "bac6c1b9f519437f82e706c280a0f13f" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38001 } }
I20260929 03:15:45.781060 21626 heartbeater.cc:503] Master 127.13.238.190:43921 was elected leader, sending a full tablet report...
I20260929 03:15:45.781350 21659 ts_tablet_manager.cc:1428] T f2fb4a9ec2de4fd8a162536a6d90d63c P bac6c1b9f519437f82e706c280a0f13f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260929 03:15:45.811384 21353 tablet.cc:2369] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:15:45.899715 21657 raft_consensus.cc:482] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:15:45.899976 21657 raft_consensus.cc:504] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4350ad5a78764d1db39c247d759ce213" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39585 } } peers { permanent_uuid: "2ae2e4cfa8764352a502fbe7ff3009c4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35993 } } peers { permanent_uuid: "bac6c1b9f519437f82e706c280a0f13f" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38001 } }
I20260929 03:15:45.901283 21657 leader_election.cc:288] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ae2e4cfa8764352a502fbe7ff3009c4 (127.13.238.130:35993), bac6c1b9f519437f82e706c280a0f13f (127.13.238.131:38001)
I20260929 03:15:45.906107 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.901955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:42173 (local address 127.13.238.131:38001)
0929 03:15:45.902075 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.902080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.902271 (+ 191us) server_negotiation.cc:407] Connection header received
0929 03:15:45.902379 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.902381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.902422 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.902491 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:45.902694 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.903411 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.904895 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.905498 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.905612 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.905715 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.905906 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.905979 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:15:45.906107 21652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.901582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:46953 (local address 127.13.238.130:35993)
0929 03:15:45.901715 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.901720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.901839 (+ 119us) server_negotiation.cc:407] Connection header received
0929 03:15:45.902024 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.902028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.902076 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.902124 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:45.902517 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.903329 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.904895 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.905498 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.905612 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.905664 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.905906 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.905979 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:15:45.906137 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.901842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:38001 (local address 127.13.238.129:42173)
0929 03:15:45.902170 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:45.902183 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:45.902302 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:45.902514 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:45.902518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:45.902534 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:45.902598 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.902602 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.903499 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.903502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.904761 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.904773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.905609 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.905612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.905756 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.905802 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 03:15:45.905876 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 03:15:45.905975 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":68,"thread_start_us":81,"threads_started":1}
I20260929 03:15:45.906137 21662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.901489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:35993 (local address 127.13.238.129:46953)
0929 03:15:45.901722 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:45.901766 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:45.901894 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:45.902290 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:45.902293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:45.902306 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:45.902407 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.902413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.903432 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.903434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.904761 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:45.904773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.905609 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.905612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:45.905765 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.905807 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 03:15:45.905876 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 03:15:45.905980 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":112,"thread_start_us":86,"threads_started":1}
I20260929 03:15:45.906792 21579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fb4a9ec2de4fd8a162536a6d90d63c" candidate_uuid: "4350ad5a78764d1db39c247d759ce213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac6c1b9f519437f82e706c280a0f13f" is_pre_election: true
I20260929 03:15:45.907049 21441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fb4a9ec2de4fd8a162536a6d90d63c" candidate_uuid: "4350ad5a78764d1db39c247d759ce213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae2e4cfa8764352a502fbe7ff3009c4" is_pre_election: true
I20260929 03:15:45.907099 21579 raft_consensus.cc:2454] T f2fb4a9ec2de4fd8a162536a6d90d63c P bac6c1b9f519437f82e706c280a0f13f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4350ad5a78764d1db39c247d759ce213 in term 0.
I20260929 03:15:45.907295 21441 raft_consensus.cc:2454] T f2fb4a9ec2de4fd8a162536a6d90d63c P 2ae2e4cfa8764352a502fbe7ff3009c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4350ad5a78764d1db39c247d759ce213 in term 0.
I20260929 03:15:45.907560 21239 leader_election.cc:302] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4350ad5a78764d1db39c247d759ce213, bac6c1b9f519437f82e706c280a0f13f; no voters:
I20260929 03:15:45.907825 21657 raft_consensus.cc:2781] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:15:45.907898 21657 raft_consensus.cc:482] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:15:45.907925 21657 raft_consensus.cc:3037] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:45.908911 21657 raft_consensus.cc:504] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4350ad5a78764d1db39c247d759ce213" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39585 } } peers { permanent_uuid: "2ae2e4cfa8764352a502fbe7ff3009c4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35993 } } peers { permanent_uuid: "bac6c1b9f519437f82e706c280a0f13f" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38001 } }
I20260929 03:15:45.909221 21657 leader_election.cc:288] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213 [CANDIDATE]: Term 1 election: Requested vote from peers 2ae2e4cfa8764352a502fbe7ff3009c4 (127.13.238.130:35993), bac6c1b9f519437f82e706c280a0f13f (127.13.238.131:38001)
I20260929 03:15:45.909538 21441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fb4a9ec2de4fd8a162536a6d90d63c" candidate_uuid: "4350ad5a78764d1db39c247d759ce213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae2e4cfa8764352a502fbe7ff3009c4"
I20260929 03:15:45.909595 21579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fb4a9ec2de4fd8a162536a6d90d63c" candidate_uuid: "4350ad5a78764d1db39c247d759ce213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac6c1b9f519437f82e706c280a0f13f"
I20260929 03:15:45.909664 21441 raft_consensus.cc:3037] T f2fb4a9ec2de4fd8a162536a6d90d63c P 2ae2e4cfa8764352a502fbe7ff3009c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:45.909718 21579 raft_consensus.cc:3037] T f2fb4a9ec2de4fd8a162536a6d90d63c P bac6c1b9f519437f82e706c280a0f13f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:45.911000 21579 raft_consensus.cc:2454] T f2fb4a9ec2de4fd8a162536a6d90d63c P bac6c1b9f519437f82e706c280a0f13f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4350ad5a78764d1db39c247d759ce213 in term 1.
I20260929 03:15:45.911000 21441 raft_consensus.cc:2454] T f2fb4a9ec2de4fd8a162536a6d90d63c P 2ae2e4cfa8764352a502fbe7ff3009c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4350ad5a78764d1db39c247d759ce213 in term 1.
I20260929 03:15:45.911262 21239 leader_election.cc:302] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4350ad5a78764d1db39c247d759ce213, bac6c1b9f519437f82e706c280a0f13f; no voters:
I20260929 03:15:45.911409 21657 raft_consensus.cc:2781] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:45.911568 21657 raft_consensus.cc:686] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213 [term 1 LEADER]: Becoming Leader. State: Replica: 4350ad5a78764d1db39c247d759ce213, State: Running, Role: LEADER
I20260929 03:15:45.911880 21657 consensus_queue.cc:237] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4350ad5a78764d1db39c247d759ce213" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39585 } } peers { permanent_uuid: "2ae2e4cfa8764352a502fbe7ff3009c4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35993 } } peers { permanent_uuid: "bac6c1b9f519437f82e706c280a0f13f" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38001 } }
I20260929 03:15:45.915112 20379 catalog_manager.cc:5495] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4350ad5a78764d1db39c247d759ce213 (127.13.238.129). New cstate: current_term: 1 leader_uuid: "4350ad5a78764d1db39c247d759ce213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4350ad5a78764d1db39c247d759ce213" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ae2e4cfa8764352a502fbe7ff3009c4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bac6c1b9f519437f82e706c280a0f13f" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38001 } health_report { overall_health: UNKNOWN } } }
I20260929 03:15:45.945511 21650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.942127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56590 (local address 127.13.238.129:39585)
0929 03:15:45.942276 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.942280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.942293 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:15:45.942442 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.942444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.942502 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.942591 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:45.942857 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.943369 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.944163 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.944416 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.944502 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.944741 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:45.945160 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:45.945248 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.945309 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.945359 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 03:15:45.947458 21628 tablet.cc:2369] T f2fb4a9ec2de4fd8a162536a6d90d63c P bac6c1b9f519437f82e706c280a0f13f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:15:45.951596 21579 raft_consensus.cc:1261] T f2fb4a9ec2de4fd8a162536a6d90d63c P bac6c1b9f519437f82e706c280a0f13f [term 1 FOLLOWER]: Refusing update from remote peer 4350ad5a78764d1db39c247d759ce213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:45.951606 21441 raft_consensus.cc:1261] T f2fb4a9ec2de4fd8a162536a6d90d63c P 2ae2e4cfa8764352a502fbe7ff3009c4 [term 1 FOLLOWER]: Refusing update from remote peer 4350ad5a78764d1db39c247d759ce213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:45.952291 21664 consensus_queue.cc:1035] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bac6c1b9f519437f82e706c280a0f13f" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:45.952471 21657 consensus_queue.cc:1035] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae2e4cfa8764352a502fbe7ff3009c4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:45.961066 21669 mvcc.cc:205] Tried to move back new op lower bound from 7334509551410601984 to 7334509551257530368. Current Snapshot: MvccSnapshot[applied={T|T < 7334509551410601984}]
I20260929 03:15:45.970476 20378 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:45.970698 20378 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:45.975569 20378 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b4bc38a6af4a493dbba8a0c697b98c77: Sending DeleteTablet for 3 replicas of tablet f2fb4a9ec2de4fd8a162536a6d90d63c
I20260929 03:15:45.976486 21284 tablet_service.cc:1514] Processing DeleteTablet for tablet f2fb4a9ec2de4fd8a162536a6d90d63c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:45 UTC) from {username='oryx', principal='oryx/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:56582
I20260929 03:15:45.976781 21421 tablet_service.cc:1514] Processing DeleteTablet for tablet f2fb4a9ec2de4fd8a162536a6d90d63c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:45 UTC) from {username='oryx', principal='oryx/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:58524
I20260929 03:15:45.977205 21678 tablet_replica.cc:330] T f2fb4a9ec2de4fd8a162536a6d90d63c P 2ae2e4cfa8764352a502fbe7ff3009c4: stopping tablet replica
I20260929 03:15:45.977540 21678 raft_consensus.cc:2229] T f2fb4a9ec2de4fd8a162536a6d90d63c P 2ae2e4cfa8764352a502fbe7ff3009c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:15:45.977931 21678 raft_consensus.cc:2258] T f2fb4a9ec2de4fd8a162536a6d90d63c P 2ae2e4cfa8764352a502fbe7ff3009c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:45.978605 21559 tablet_service.cc:1514] Processing DeleteTablet for tablet f2fb4a9ec2de4fd8a162536a6d90d63c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:45 UTC) from {username='oryx', principal='oryx/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:37362
I20260929 03:15:45.979084 21680 tablet_replica.cc:330] T f2fb4a9ec2de4fd8a162536a6d90d63c P bac6c1b9f519437f82e706c280a0f13f: stopping tablet replica
I20260929 03:15:45.979388 21678 ts_tablet_manager.cc:1905] T f2fb4a9ec2de4fd8a162536a6d90d63c P 2ae2e4cfa8764352a502fbe7ff3009c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:45.979400 21680 raft_consensus.cc:2229] T f2fb4a9ec2de4fd8a162536a6d90d63c P bac6c1b9f519437f82e706c280a0f13f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:15:45.979769 21680 raft_consensus.cc:2258] T f2fb4a9ec2de4fd8a162536a6d90d63c P bac6c1b9f519437f82e706c280a0f13f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:45.980010 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 20347
I20260929 03:15:45.981432 21680 ts_tablet_manager.cc:1905] T f2fb4a9ec2de4fd8a162536a6d90d63c P bac6c1b9f519437f82e706c280a0f13f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:45.982012 21678 ts_tablet_manager.cc:1918] T f2fb4a9ec2de4fd8a162536a6d90d63c P 2ae2e4cfa8764352a502fbe7ff3009c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:15:45.982105 21678 log.cc:1196] T f2fb4a9ec2de4fd8a162536a6d90d63c P 2ae2e4cfa8764352a502fbe7ff3009c4: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/wal/wals/f2fb4a9ec2de4fd8a162536a6d90d63c
I20260929 03:15:45.984398 21680 ts_tablet_manager.cc:1918] T f2fb4a9ec2de4fd8a162536a6d90d63c P bac6c1b9f519437f82e706c280a0f13f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:15:45.984557 21680 log.cc:1196] T f2fb4a9ec2de4fd8a162536a6d90d63c P bac6c1b9f519437f82e706c280a0f13f: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/f2fb4a9ec2de4fd8a162536a6d90d63c
I20260929 03:15:45.984926 21680 ts_tablet_manager.cc:1939] T f2fb4a9ec2de4fd8a162536a6d90d63c P bac6c1b9f519437f82e706c280a0f13f: Deleting consensus metadata
I20260929 03:15:45.986472 21678 ts_tablet_manager.cc:1939] T f2fb4a9ec2de4fd8a162536a6d90d63c P 2ae2e4cfa8764352a502fbe7ff3009c4: Deleting consensus metadata
W20260929 03:15:45.988341 21512 connection.cc:535] server connection from 127.0.0.1:37362 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 03:15:45.988713 21377 connection.cc:535] server connection from 127.0.0.1:58524 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 03:15:45.989787 21677 tablet_replica.cc:330] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213: stopping tablet replica
I20260929 03:15:45.990100 21677 raft_consensus.cc:2229] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:15:45.990535 21677 raft_consensus.cc:2258] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:45.990705 21639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.978165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52554 (local address 127.13.238.189:46533)
0929 03:15:45.978290 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.978295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.978307 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:15:45.978344 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.978347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.978391 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.978465 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:45.983152 (+ 4687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.983911 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.985064 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.985399 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.985475 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.987375 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:45.987379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:45.987382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:45.987409 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:45.988896 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.989480 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.989484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.989485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.989570 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.989820 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.989823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.989825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.990022 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:45.990141 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.990436 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.990551 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:15:45.993911 21677 ts_tablet_manager.cc:1905] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:45.997073 21677 ts_tablet_manager.cc:1918] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:15:45.997184 21677 log.cc:1196] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-0/wal/wals/f2fb4a9ec2de4fd8a162536a6d90d63c
I20260929 03:15:45.997479 21677 ts_tablet_manager.cc:1939] T f2fb4a9ec2de4fd8a162536a6d90d63c P 4350ad5a78764d1db39c247d759ce213: Deleting consensus metadata
W20260929 03:15:45.998606 21237 connection.cc:441] server connection from 127.0.0.1:56582 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56582 (request call id 1) could send its response
I20260929 03:15:45.998936 21354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.978131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49180 (local address 127.13.238.188:44881)
0929 03:15:45.978315 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.978320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.990833 (+ 12513us) server_negotiation.cc:407] Connection header received
0929 03:15:45.991010 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.991014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.991065 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.991116 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:45.991905 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.992539 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.993509 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.993789 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.993874 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:45.995322 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:45.995325 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:45.995328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:45.995350 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:45.996894 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.997344 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.997348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.997350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.997430 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:45.997727 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:45.997731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:45.997733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:45.997942 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:45.998041 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:45.998670 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 03:15:45.998778 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260929 03:15:46.003769 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.992843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49196 (local address 127.13.238.188:44881)
0929 03:15:45.992951 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.992955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.995142 (+ 2187us) server_negotiation.cc:407] Connection header received
0929 03:15:45.995301 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.995304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.995346 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.995393 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:45.996722 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.997444 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.998650 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:45.998968 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:45.999040 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:46.000900 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:46.000904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:46.000907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:46.000930 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:46.002332 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:46.002679 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:46.002683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:46.002685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:46.002757 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:46.002993 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:46.002996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:46.002998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:46.003160 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:46.003274 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:46.003533 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 03:15:46.003653 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:15:46.007567 21639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:45.994820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52560 (local address 127.13.238.189:46533)
0929 03:15:45.994922 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:45.994926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:45.999223 (+ 4297us) server_negotiation.cc:407] Connection header received
0929 03:15:45.999382 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:45.999385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:45.999433 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:45.999479 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:46.000366 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:46.001091 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:46.002556 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:46.002889 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:46.002960 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:46.004474 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:46.004478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:46.004481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:46.004507 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:46.006065 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:46.006497 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:46.006500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:46.006502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:46.006586 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:46.006816 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:46.006819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:46.006821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:46.007018 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:46.007145 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:46.007286 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 03:15:46.007408 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:15:46.963527 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:46.962975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.13.238.131:40525)
0929 03:15:46.963267 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:46.963404 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":91,"threads_started":1}
W20260929 03:15:46.964246 21626 heartbeater.cc:644] Failed to heartbeat to 127.13.238.190:43921 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
I20260929 03:15:46.965200 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:46.964817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.13.238.130:38375)
0929 03:15:46.965025 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:46.965111 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":65,"threads_started":1}
W20260929 03:15:46.965473 21488 heartbeater.cc:644] Failed to heartbeat to 127.13.238.190:43921 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
I20260929 03:15:46.966097 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:46.965738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.13.238.129:45713)
0929 03:15:46.965945 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:46.966023 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":64,"threads_started":1}
W20260929 03:15:46.966348 21351 heartbeater.cc:644] Failed to heartbeat to 127.13.238.190:43921 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
I20260929 03:15:47.475515 21691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b4bc38a6af4a493dbba8a0c697b98c77)
I20260929 03:15:47.475698 21691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: false } }
I20260929 03:15:47.476459 21692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b4bc38a6af4a493dbba8a0c697b98c77)
I20260929 03:15:47.476619 21691 leader_election.cc:288] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b4bc38a6af4a493dbba8a0c697b98c77 (127.13.238.190:43921), 12996e1d57454853ba637b4f8660e902 (127.13.238.189:46533)
I20260929 03:15:47.476637 21692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: false } }
I20260929 03:15:47.477464 21693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.476800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.0.0.1:43742)
0929 03:15:47.477277 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.477362 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":65,"threads_started":1}
W20260929 03:15:47.477680 21156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
I20260929 03:15:47.478078 21696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.477661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.0.0.1:43756)
0929 03:15:47.477915 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.477989 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":80,"threads_started":1}
W20260929 03:15:47.478366 20965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
I20260929 03:15:47.478377 21692 leader_election.cc:288] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b4bc38a6af4a493dbba8a0c697b98c77 (127.13.238.190:43921), 02939efbf00847fa8d405999eb01407e (127.13.238.188:44881)
I20260929 03:15:47.478880 21693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.478334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.0.0.1:43768)
0929 03:15:47.478766 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.478814 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375}
W20260929 03:15:47.478998 21156 leader_election.cc:334] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b4bc38a6af4a493dbba8a0c697b98c77 (127.13.238.190:43921): Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
I20260929 03:15:47.479396 21701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.478983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.0.0.1:43774)
0929 03:15:47.479272 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.479321 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":74,"threads_started":1}
W20260929 03:15:47.479496 20965 leader_election.cc:334] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b4bc38a6af4a493dbba8a0c697b98c77 (127.13.238.190:43921): Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
I20260929 03:15:47.481122 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.477115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:46533 (local address 127.0.0.1:52574)
0929 03:15:47.477316 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.477328 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:47.477422 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:47.477856 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:47.477859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:47.477880 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:47.477981 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:47.477989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:47.479002 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:47.479006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:47.480135 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:47.480146 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:47.480755 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:47.480759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:47.480848 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:47.480875 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 03:15:47.480953 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:15:47.481027 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":23,"thread_start_us":49,"threads_started":1}
I20260929 03:15:47.481150 21695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.477204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52574 (local address 127.13.238.189:46533)
0929 03:15:47.477439 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:47.477443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:47.477492 (+ 49us) server_negotiation.cc:407] Connection header received
0929 03:15:47.477538 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:47.477540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:47.477587 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:47.477680 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:47.478095 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:47.478885 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:47.480247 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:47.480646 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:47.480763 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:47.480805 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:47.480972 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 03:15:47.481043 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":56,"threads_started":1}
I20260929 03:15:47.481575 20990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02939efbf00847fa8d405999eb01407e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "12996e1d57454853ba637b4f8660e902" is_pre_election: true
I20260929 03:15:47.481775 20990 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02939efbf00847fa8d405999eb01407e in term 1.
I20260929 03:15:47.482041 21698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.478085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:44881 (local address 127.0.0.1:49212)
0929 03:15:47.478563 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.478576 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:47.478652 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:47.478947 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:47.478950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:47.478968 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:47.479060 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:47.479067 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:47.479967 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:47.479971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:47.480896 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:47.480933 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:47.481662 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:47.481667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:47.481751 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:47.481775 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 03:15:47.481851 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 03:15:47.481920 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":1,"thread_start_us":51,"threads_started":1}
I20260929 03:15:47.482125 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.478436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49212 (local address 127.13.238.188:44881)
0929 03:15:47.478708 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:47.478716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:47.478742 (+ 26us) server_negotiation.cc:407] Connection header received
0929 03:15:47.478783 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:47.478785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:47.478825 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:47.479161 (+ 336us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:47.479282 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:47.479867 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:47.481027 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:47.481557 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:47.481645 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:47.481681 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:47.481879 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 03:15:47.481948 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":214,"thread_start_us":90,"threads_started":1}
I20260929 03:15:47.482148 21156 leader_election.cc:302] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02939efbf00847fa8d405999eb01407e, 12996e1d57454853ba637b4f8660e902; no voters: b4bc38a6af4a493dbba8a0c697b98c77
I20260929 03:15:47.482395 21691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 03:15:47.482450 21691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [term 1 FOLLOWER]: Starting leader election (detected failure of leader b4bc38a6af4a493dbba8a0c697b98c77)
I20260929 03:15:47.482491 21691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:15:47.482743 21181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12996e1d57454853ba637b4f8660e902" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "02939efbf00847fa8d405999eb01407e" is_pre_election: true
I20260929 03:15:47.483575 21691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: false } }
I20260929 03:15:47.483794 21181 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 12996e1d57454853ba637b4f8660e902 in current term 2: Already voted for candidate 02939efbf00847fa8d405999eb01407e in this term.
I20260929 03:15:47.483896 21691 leader_election.cc:288] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [CANDIDATE]: Term 2 election: Requested vote from peers b4bc38a6af4a493dbba8a0c697b98c77 (127.13.238.190:43921), 12996e1d57454853ba637b4f8660e902 (127.13.238.189:46533)
I20260929 03:15:47.484220 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.484025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.0.0.1:43782)
0929 03:15:47.484114 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.484164 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260929 03:15:47.484355 20990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02939efbf00847fa8d405999eb01407e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "12996e1d57454853ba637b4f8660e902"
I20260929 03:15:47.484431 20966 leader_election.cc:302] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 12996e1d57454853ba637b4f8660e902; no voters: 02939efbf00847fa8d405999eb01407e, b4bc38a6af4a493dbba8a0c697b98c77
I20260929 03:15:47.484457 20990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:15:47.485047 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.484844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.0.0.1:43792)
0929 03:15:47.484944 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.484993 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 03:15:47.485147 21156 leader_election.cc:334] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b4bc38a6af4a493dbba8a0c697b98c77 (127.13.238.190:43921): Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
I20260929 03:15:47.485549 20990 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02939efbf00847fa8d405999eb01407e in term 2.
I20260929 03:15:47.485646 21692 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:15:47.485845 21156 leader_election.cc:302] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02939efbf00847fa8d405999eb01407e, 12996e1d57454853ba637b4f8660e902; no voters: b4bc38a6af4a493dbba8a0c697b98c77
I20260929 03:15:47.486027 21691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:15:47.486236 21691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [term 2 LEADER]: Becoming Leader. State: Replica: 02939efbf00847fa8d405999eb01407e, State: Running, Role: LEADER
I20260929 03:15:47.486475 21691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: false } }
I20260929 03:15:47.487367 21702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02939efbf00847fa8d405999eb01407e. Latest consensus state: current_term: 2 leader_uuid: "02939efbf00847fa8d405999eb01407e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: false } } }
I20260929 03:15:47.487481 21702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:47.487764 21704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:15:47.489810 21704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:15:47.490242 21704 catalog_manager.cc:1259] Loaded cluster ID: e30d83aca1b1425e8a563e680cc2740b
I20260929 03:15:47.490288 21704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:15:47.490928 21704 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:15:47.491297 21704 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e: Loaded TSK: 0
I20260929 03:15:47.492028 21704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:15:47.523983 21171 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:15:47.527045 21171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:15:47.529956 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.529655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.0.0.1:43814)
0929 03:15:47.529810 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.529874 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 03:15:47.529942 20990 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [term 2 FOLLOWER]: Refusing update from remote peer 02939efbf00847fa8d405999eb01407e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 03:15:47.530426 21702 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [LEADER]: Connected to new peer: Peer: permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 03:15:47.530745 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.530485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.0.0.1:43818)
0929 03:15:47.530591 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.530638 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 03:15:47.530886 21156 consensus_peers.cc:487] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e -> Peer b4bc38a6af4a493dbba8a0c697b98c77 (127.13.238.190:43921): Couldn't send request to peer b4bc38a6af4a493dbba8a0c697b98c77. Status: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 03:15:47.531833 21692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02939efbf00847fa8d405999eb01407e. Latest consensus state: current_term: 2 leader_uuid: "02939efbf00847fa8d405999eb01407e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: false } } }
I20260929 03:15:47.531947 21692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:15:47.532961 21702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "02939efbf00847fa8d405999eb01407e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: false } } }
I20260929 03:15:47.533072 21702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:47.533384 21692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "02939efbf00847fa8d405999eb01407e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b4bc38a6af4a493dbba8a0c697b98c77" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43921 } } peers { permanent_uuid: "12996e1d57454853ba637b4f8660e902" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46533 } attrs { promote: false } } peers { permanent_uuid: "02939efbf00847fa8d405999eb01407e" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 44881 } attrs { promote: false } } }
I20260929 03:15:47.533499 21692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12996e1d57454853ba637b4f8660e902 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:15:47.542953 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.538708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:38001 (local address 127.0.0.1:37364)
0929 03:15:47.538801 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.538834 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:47.538912 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:47.539423 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:47.539427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:47.539446 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:47.539542 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:47.539548 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:47.540558 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:47.540561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:47.541651 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:47.541660 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:47.542581 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:47.542585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:47.542670 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:47.542718 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 03:15:47.542762 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 03:15:47.542827 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 03:15:47.543100 21693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.538987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:35993 (local address 127.0.0.1:58528)
0929 03:15:47.539071 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.539085 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:47.539162 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:47.539786 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:47.539789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:47.539804 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:47.539901 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:47.539906 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:47.540908 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:47.540912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:47.541763 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:47.541771 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:47.542678 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:47.542691 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:47.542823 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:47.542843 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 03:15:47.542908 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 03:15:47.542979 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":20}
I20260929 03:15:47.543135 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.539301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58528 (local address 127.13.238.130:35993)
0929 03:15:47.539550 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:47.539555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:47.539568 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:15:47.539618 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:47.539621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:47.539665 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:47.539985 (+ 320us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:47.540131 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:47.540813 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:47.541874 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:47.542356 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:47.542468 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:47.542522 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:47.542938 (+ 416us) server_negotiation.cc:299] Negotiation successful
0929 03:15:47.543013 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":70,"threads_started":1}
I20260929 03:15:47.543509 21711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.538995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37364 (local address 127.13.238.131:38001)
0929 03:15:47.539231 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:47.539235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:47.539246 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:15:47.539288 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:47.539290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:47.539326 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:47.539392 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:47.539676 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:47.540469 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:47.541793 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:47.542458 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:47.543221 (+ 763us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:47.543279 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:47.543312 (+ 33us) server_negotiation.cc:299] Negotiation successful
0929 03:15:47.543384 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
I20260929 03:15:47.543977 21559 tablet_service.cc:1467] Processing CreateTablet for tablet 97df4001b50a46679e7438e6b429d401 (DEFAULT_TABLE table=test-table [id=52b1d1a3f2ea4ab397ba36c38df31d3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:47.544252 21713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.539779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:39585 (local address 127.0.0.1:56596)
0929 03:15:47.540270 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.540281 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:47.540355 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:47.540729 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:47.540732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:47.540750 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:47.540898 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:47.540907 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:47.541947 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:47.541951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:47.543025 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:47.543034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:47.543838 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:47.543841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:47.543949 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:47.543973 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 03:15:47.544043 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 03:15:47.544116 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":1,"thread_start_us":89,"threads_started":1}
I20260929 03:15:47.544262 21559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97df4001b50a46679e7438e6b429d401. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:47.544381 21421 tablet_service.cc:1467] Processing CreateTablet for tablet 97df4001b50a46679e7438e6b429d401 (DEFAULT_TABLE table=test-table [id=52b1d1a3f2ea4ab397ba36c38df31d3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:47.544618 21421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97df4001b50a46679e7438e6b429d401. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:47.546509 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.540130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56596 (local address 127.13.238.129:39585)
0929 03:15:47.540517 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:47.540521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:47.540534 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:15:47.540580 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:47.540583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:47.540627 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:47.540718 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:47.541047 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:47.541798 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:47.543152 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:47.543724 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:47.543838 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:47.543884 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:47.546316 (+ 2432us) server_negotiation.cc:299] Negotiation successful
0929 03:15:47.546403 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":69,"threads_started":1}
I20260929 03:15:47.546972 21715 tablet_bootstrap.cc:492] T 97df4001b50a46679e7438e6b429d401 P bac6c1b9f519437f82e706c280a0f13f: Bootstrap starting.
I20260929 03:15:47.547363 21284 tablet_service.cc:1467] Processing CreateTablet for tablet 97df4001b50a46679e7438e6b429d401 (DEFAULT_TABLE table=test-table [id=52b1d1a3f2ea4ab397ba36c38df31d3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:47.547612 21284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97df4001b50a46679e7438e6b429d401. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:47.548003 21715 tablet_bootstrap.cc:654] T 97df4001b50a46679e7438e6b429d401 P bac6c1b9f519437f82e706c280a0f13f: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:47.548606 21716 tablet_bootstrap.cc:492] T 97df4001b50a46679e7438e6b429d401 P 2ae2e4cfa8764352a502fbe7ff3009c4: Bootstrap starting.
I20260929 03:15:47.549166 21715 tablet_bootstrap.cc:492] T 97df4001b50a46679e7438e6b429d401 P bac6c1b9f519437f82e706c280a0f13f: No bootstrap required, opened a new log
I20260929 03:15:47.549265 21715 ts_tablet_manager.cc:1397] T 97df4001b50a46679e7438e6b429d401 P bac6c1b9f519437f82e706c280a0f13f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:15:47.549391 21716 tablet_bootstrap.cc:654] T 97df4001b50a46679e7438e6b429d401 P 2ae2e4cfa8764352a502fbe7ff3009c4: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:47.549702 21715 raft_consensus.cc:348] T 97df4001b50a46679e7438e6b429d401 P bac6c1b9f519437f82e706c280a0f13f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac6c1b9f519437f82e706c280a0f13f" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38001 } } peers { permanent_uuid: "2ae2e4cfa8764352a502fbe7ff3009c4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35993 } } peers { permanent_uuid: "4350ad5a78764d1db39c247d759ce213" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39585 } }
I20260929 03:15:47.549825 21715 raft_consensus.cc:374] T 97df4001b50a46679e7438e6b429d401 P bac6c1b9f519437f82e706c280a0f13f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:47.549875 21715 raft_consensus.cc:729] T 97df4001b50a46679e7438e6b429d401 P bac6c1b9f519437f82e706c280a0f13f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac6c1b9f519437f82e706c280a0f13f, State: Initialized, Role: FOLLOWER
I20260929 03:15:47.550000 21715 consensus_queue.cc:260] T 97df4001b50a46679e7438e6b429d401 P bac6c1b9f519437f82e706c280a0f13f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac6c1b9f519437f82e706c280a0f13f" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38001 } } peers { permanent_uuid: "2ae2e4cfa8764352a502fbe7ff3009c4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35993 } } peers { permanent_uuid: "4350ad5a78764d1db39c247d759ce213" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39585 } }
I20260929 03:15:47.550387 21715 ts_tablet_manager.cc:1428] T 97df4001b50a46679e7438e6b429d401 P bac6c1b9f519437f82e706c280a0f13f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:47.550549 21719 tablet_bootstrap.cc:492] T 97df4001b50a46679e7438e6b429d401 P 4350ad5a78764d1db39c247d759ce213: Bootstrap starting.
I20260929 03:15:47.550614 21716 tablet_bootstrap.cc:492] T 97df4001b50a46679e7438e6b429d401 P 2ae2e4cfa8764352a502fbe7ff3009c4: No bootstrap required, opened a new log
I20260929 03:15:47.550700 21716 ts_tablet_manager.cc:1397] T 97df4001b50a46679e7438e6b429d401 P 2ae2e4cfa8764352a502fbe7ff3009c4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:15:47.551296 21716 raft_consensus.cc:348] T 97df4001b50a46679e7438e6b429d401 P 2ae2e4cfa8764352a502fbe7ff3009c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac6c1b9f519437f82e706c280a0f13f" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38001 } } peers { permanent_uuid: "2ae2e4cfa8764352a502fbe7ff3009c4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35993 } } peers { permanent_uuid: "4350ad5a78764d1db39c247d759ce213" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39585 } }
I20260929 03:15:47.551465 21716 raft_consensus.cc:374] T 97df4001b50a46679e7438e6b429d401 P 2ae2e4cfa8764352a502fbe7ff3009c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:47.551482 21719 tablet_bootstrap.cc:654] T 97df4001b50a46679e7438e6b429d401 P 4350ad5a78764d1db39c247d759ce213: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:47.551507 21716 raft_consensus.cc:729] T 97df4001b50a46679e7438e6b429d401 P 2ae2e4cfa8764352a502fbe7ff3009c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae2e4cfa8764352a502fbe7ff3009c4, State: Initialized, Role: FOLLOWER
I20260929 03:15:47.551677 21716 consensus_queue.cc:260] T 97df4001b50a46679e7438e6b429d401 P 2ae2e4cfa8764352a502fbe7ff3009c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac6c1b9f519437f82e706c280a0f13f" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38001 } } peers { permanent_uuid: "2ae2e4cfa8764352a502fbe7ff3009c4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35993 } } peers { permanent_uuid: "4350ad5a78764d1db39c247d759ce213" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39585 } }
I20260929 03:15:47.552006 21716 ts_tablet_manager.cc:1428] T 97df4001b50a46679e7438e6b429d401 P 2ae2e4cfa8764352a502fbe7ff3009c4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 03:15:47.553030 21719 tablet_bootstrap.cc:492] T 97df4001b50a46679e7438e6b429d401 P 4350ad5a78764d1db39c247d759ce213: No bootstrap required, opened a new log
I20260929 03:15:47.553066 21724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.552525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.13.238.130:43723)
0929 03:15:47.552902 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.552962 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":110,"threads_started":1}
I20260929 03:15:47.553102 21719 ts_tablet_manager.cc:1397] T 97df4001b50a46679e7438e6b429d401 P 4350ad5a78764d1db39c247d759ce213: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 03:15:47.553495 21719 raft_consensus.cc:348] T 97df4001b50a46679e7438e6b429d401 P 4350ad5a78764d1db39c247d759ce213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac6c1b9f519437f82e706c280a0f13f" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38001 } } peers { permanent_uuid: "2ae2e4cfa8764352a502fbe7ff3009c4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35993 } } peers { permanent_uuid: "4350ad5a78764d1db39c247d759ce213" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39585 } }
I20260929 03:15:47.553608 21719 raft_consensus.cc:374] T 97df4001b50a46679e7438e6b429d401 P 4350ad5a78764d1db39c247d759ce213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:47.553642 21719 raft_consensus.cc:729] T 97df4001b50a46679e7438e6b429d401 P 4350ad5a78764d1db39c247d759ce213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4350ad5a78764d1db39c247d759ce213, State: Initialized, Role: FOLLOWER
I20260929 03:15:47.553754 21719 consensus_queue.cc:260] T 97df4001b50a46679e7438e6b429d401 P 4350ad5a78764d1db39c247d759ce213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac6c1b9f519437f82e706c280a0f13f" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38001 } } peers { permanent_uuid: "2ae2e4cfa8764352a502fbe7ff3009c4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35993 } } peers { permanent_uuid: "4350ad5a78764d1db39c247d759ce213" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39585 } }
I20260929 03:15:47.553802 21625 heartbeater.cc:503] Master 127.13.238.188:44881 was elected leader, sending a full tablet report...
I20260929 03:15:47.554065 21719 ts_tablet_manager.cc:1428] T 97df4001b50a46679e7438e6b429d401 P 4350ad5a78764d1db39c247d759ce213: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:47.555557 21721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.550798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.13.238.131:47371)
0929 03:15:47.555381 (+ 4583us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.555450 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4511,"thread_start_us":86,"threads_started":1}
I20260929 03:15:47.555840 21487 heartbeater.cc:503] Master 127.13.238.188:44881 was elected leader, sending a full tablet report...
I20260929 03:15:47.556159 21350 heartbeater.cc:503] Master 127.13.238.188:44881 was elected leader, sending a full tablet report...
I20260929 03:15:47.556152 21726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.554521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.13.238.129:47791)
0929 03:15:47.556009 (+ 1488us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.556065 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1428,"thread_start_us":85,"threads_started":1}
I20260929 03:15:47.557454 21721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.556854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.13.238.131:46125)
0929 03:15:47.557323 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.557387 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396}
I20260929 03:15:47.557780 21726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.556898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.13.238.129:60235)
0929 03:15:47.557651 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.557714 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":661}
I20260929 03:15:47.558320 21724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.557255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.13.238.130:42325)
0929 03:15:47.558176 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.558256 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":845}
I20260929 03:15:47.631662 21725 raft_consensus.cc:482] T 97df4001b50a46679e7438e6b429d401 P 4350ad5a78764d1db39c247d759ce213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:15:47.631810 21725 raft_consensus.cc:504] T 97df4001b50a46679e7438e6b429d401 P 4350ad5a78764d1db39c247d759ce213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac6c1b9f519437f82e706c280a0f13f" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38001 } } peers { permanent_uuid: "2ae2e4cfa8764352a502fbe7ff3009c4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35993 } } peers { permanent_uuid: "4350ad5a78764d1db39c247d759ce213" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39585 } }
I20260929 03:15:47.632252 21725 leader_election.cc:288] T 97df4001b50a46679e7438e6b429d401 P 4350ad5a78764d1db39c247d759ce213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bac6c1b9f519437f82e706c280a0f13f (127.13.238.131:38001), 2ae2e4cfa8764352a502fbe7ff3009c4 (127.13.238.130:35993)
I20260929 03:15:47.632814 21579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97df4001b50a46679e7438e6b429d401" candidate_uuid: "4350ad5a78764d1db39c247d759ce213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac6c1b9f519437f82e706c280a0f13f" is_pre_election: true
I20260929 03:15:47.632823 21441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97df4001b50a46679e7438e6b429d401" candidate_uuid: "4350ad5a78764d1db39c247d759ce213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae2e4cfa8764352a502fbe7ff3009c4" is_pre_election: true
I20260929 03:15:47.632951 21579 raft_consensus.cc:2454] T 97df4001b50a46679e7438e6b429d401 P bac6c1b9f519437f82e706c280a0f13f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4350ad5a78764d1db39c247d759ce213 in term 0.
I20260929 03:15:47.632972 21441 raft_consensus.cc:2454] T 97df4001b50a46679e7438e6b429d401 P 2ae2e4cfa8764352a502fbe7ff3009c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4350ad5a78764d1db39c247d759ce213 in term 0.
I20260929 03:15:47.633279 21239 leader_election.cc:302] T 97df4001b50a46679e7438e6b429d401 P 4350ad5a78764d1db39c247d759ce213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4350ad5a78764d1db39c247d759ce213, bac6c1b9f519437f82e706c280a0f13f; no voters:
I20260929 03:15:47.633467 21725 raft_consensus.cc:2781] T 97df4001b50a46679e7438e6b429d401 P 4350ad5a78764d1db39c247d759ce213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:15:47.633522 21725 raft_consensus.cc:482] T 97df4001b50a46679e7438e6b429d401 P 4350ad5a78764d1db39c247d759ce213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:15:47.633548 21725 raft_consensus.cc:3037] T 97df4001b50a46679e7438e6b429d401 P 4350ad5a78764d1db39c247d759ce213 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:47.634575 21725 raft_consensus.cc:504] T 97df4001b50a46679e7438e6b429d401 P 4350ad5a78764d1db39c247d759ce213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac6c1b9f519437f82e706c280a0f13f" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38001 } } peers { permanent_uuid: "2ae2e4cfa8764352a502fbe7ff3009c4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35993 } } peers { permanent_uuid: "4350ad5a78764d1db39c247d759ce213" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39585 } }
I20260929 03:15:47.634878 21725 leader_election.cc:288] T 97df4001b50a46679e7438e6b429d401 P 4350ad5a78764d1db39c247d759ce213 [CANDIDATE]: Term 1 election: Requested vote from peers bac6c1b9f519437f82e706c280a0f13f (127.13.238.131:38001), 2ae2e4cfa8764352a502fbe7ff3009c4 (127.13.238.130:35993)
I20260929 03:15:47.635222 21579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97df4001b50a46679e7438e6b429d401" candidate_uuid: "4350ad5a78764d1db39c247d759ce213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac6c1b9f519437f82e706c280a0f13f"
I20260929 03:15:47.635334 21579 raft_consensus.cc:3037] T 97df4001b50a46679e7438e6b429d401 P bac6c1b9f519437f82e706c280a0f13f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:47.635342 21441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97df4001b50a46679e7438e6b429d401" candidate_uuid: "4350ad5a78764d1db39c247d759ce213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae2e4cfa8764352a502fbe7ff3009c4"
I20260929 03:15:47.635476 21441 raft_consensus.cc:3037] T 97df4001b50a46679e7438e6b429d401 P 2ae2e4cfa8764352a502fbe7ff3009c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:47.636185 21579 raft_consensus.cc:2454] T 97df4001b50a46679e7438e6b429d401 P bac6c1b9f519437f82e706c280a0f13f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4350ad5a78764d1db39c247d759ce213 in term 1.
I20260929 03:15:47.636399 21441 raft_consensus.cc:2454] T 97df4001b50a46679e7438e6b429d401 P 2ae2e4cfa8764352a502fbe7ff3009c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4350ad5a78764d1db39c247d759ce213 in term 1.
I20260929 03:15:47.636431 21239 leader_election.cc:302] T 97df4001b50a46679e7438e6b429d401 P 4350ad5a78764d1db39c247d759ce213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4350ad5a78764d1db39c247d759ce213, bac6c1b9f519437f82e706c280a0f13f; no voters:
I20260929 03:15:47.636566 21725 raft_consensus.cc:2781] T 97df4001b50a46679e7438e6b429d401 P 4350ad5a78764d1db39c247d759ce213 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:47.636814 21725 raft_consensus.cc:686] T 97df4001b50a46679e7438e6b429d401 P 4350ad5a78764d1db39c247d759ce213 [term 1 LEADER]: Becoming Leader. State: Replica: 4350ad5a78764d1db39c247d759ce213, State: Running, Role: LEADER
I20260929 03:15:47.637156 21725 consensus_queue.cc:237] T 97df4001b50a46679e7438e6b429d401 P 4350ad5a78764d1db39c247d759ce213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac6c1b9f519437f82e706c280a0f13f" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38001 } } peers { permanent_uuid: "2ae2e4cfa8764352a502fbe7ff3009c4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35993 } } peers { permanent_uuid: "4350ad5a78764d1db39c247d759ce213" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39585 } }
I20260929 03:15:47.637527 21726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.637271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.13.238.129:53253)
0929 03:15:47.637381 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.637445 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 03:15:47.638955 21169 catalog_manager.cc:5495] T 97df4001b50a46679e7438e6b429d401 P 4350ad5a78764d1db39c247d759ce213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4350ad5a78764d1db39c247d759ce213 (127.13.238.129). New cstate: current_term: 1 leader_uuid: "4350ad5a78764d1db39c247d759ce213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac6c1b9f519437f82e706c280a0f13f" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae2e4cfa8764352a502fbe7ff3009c4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4350ad5a78764d1db39c247d759ce213" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39585 } health_report { overall_health: HEALTHY } } }
I20260929 03:15:47.668864 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.666174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56606 (local address 127.13.238.129:39585)
0929 03:15:47.666304 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:47.666308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:47.666321 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:15:47.666408 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:47.666410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:47.666453 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:47.666512 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:47.666685 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:47.667182 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:47.668062 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:47.668286 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:47.668363 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:47.668541 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:47.668603 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:47.668645 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:47.668716 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 03:15:47.668759 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:15:47.671041 21441 raft_consensus.cc:1261] T 97df4001b50a46679e7438e6b429d401 P 2ae2e4cfa8764352a502fbe7ff3009c4 [term 1 FOLLOWER]: Refusing update from remote peer 4350ad5a78764d1db39c247d759ce213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:47.671082 21579 raft_consensus.cc:1261] T 97df4001b50a46679e7438e6b429d401 P bac6c1b9f519437f82e706c280a0f13f [term 1 FOLLOWER]: Refusing update from remote peer 4350ad5a78764d1db39c247d759ce213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:47.671471 21727 consensus_queue.cc:1035] T 97df4001b50a46679e7438e6b429d401 P 4350ad5a78764d1db39c247d759ce213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bac6c1b9f519437f82e706c280a0f13f" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 38001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:47.671644 21725 consensus_queue.cc:1035] T 97df4001b50a46679e7438e6b429d401 P 4350ad5a78764d1db39c247d759ce213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae2e4cfa8764352a502fbe7ff3009c4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:47.673489 21724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.672841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.13.238.130:59455)
0929 03:15:47.673353 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.673414 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442}
I20260929 03:15:47.675127 21721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.674761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.13.238.131:37861)
0929 03:15:47.675005 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.675057 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260929 03:15:47.675815 21724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.675602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.13.238.130:33231)
0929 03:15:47.675698 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.675755 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 03:15:47.678539 21726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.678223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.13.238.129:46451)
0929 03:15:47.678425 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.678475 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260929 03:15:47.679188 21726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.678983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.13.238.129:51837)
0929 03:15:47.679079 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.679128 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 03:15:47.679704 21170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:47.679847 21170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:47.680384 21732 mvcc.cc:205] Tried to move back new op lower bound from 7334509558456164352 to 7334509558323408896. Current Snapshot: MvccSnapshot[applied={T|T < 7334509558456164352}]
I20260929 03:15:47.680722 21721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.680517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43921 (local address 127.13.238.131:42851)
0929 03:15:47.680605 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.680654 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 03:15:47.683586 21170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02939efbf00847fa8d405999eb01407e: Sending DeleteTablet for 3 replicas of tablet 97df4001b50a46679e7438e6b429d401
I20260929 03:15:47.684204 21421 tablet_service.cc:1514] Processing DeleteTablet for tablet 97df4001b50a46679e7438e6b429d401 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:47 UTC) from {username='oryx', principal='oryx/127.13.238.188@KRBTEST.COM'} at 127.0.0.1:58528
I20260929 03:15:47.684234 21284 tablet_service.cc:1514] Processing DeleteTablet for tablet 97df4001b50a46679e7438e6b429d401 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:47 UTC) from {username='oryx', principal='oryx/127.13.238.188@KRBTEST.COM'} at 127.0.0.1:56596
I20260929 03:15:47.684263 21559 tablet_service.cc:1514] Processing DeleteTablet for tablet 97df4001b50a46679e7438e6b429d401 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:47 UTC) from {username='oryx', principal='oryx/127.13.238.188@KRBTEST.COM'} at 127.0.0.1:37364
I20260929 03:15:47.684640 21739 tablet_replica.cc:330] T 97df4001b50a46679e7438e6b429d401 P 2ae2e4cfa8764352a502fbe7ff3009c4: stopping tablet replica
I20260929 03:15:47.684764 21739 raft_consensus.cc:2229] T 97df4001b50a46679e7438e6b429d401 P 2ae2e4cfa8764352a502fbe7ff3009c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:15:47.684814 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 21124
I20260929 03:15:47.684885 21739 raft_consensus.cc:2258] T 97df4001b50a46679e7438e6b429d401 P 2ae2e4cfa8764352a502fbe7ff3009c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:47.686357 21740 tablet_replica.cc:330] T 97df4001b50a46679e7438e6b429d401 P bac6c1b9f519437f82e706c280a0f13f: stopping tablet replica
I20260929 03:15:47.686460 21740 raft_consensus.cc:2229] T 97df4001b50a46679e7438e6b429d401 P bac6c1b9f519437f82e706c280a0f13f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:15:47.686558 21740 raft_consensus.cc:2258] T 97df4001b50a46679e7438e6b429d401 P bac6c1b9f519437f82e706c280a0f13f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:47.686759 21739 ts_tablet_manager.cc:1905] T 97df4001b50a46679e7438e6b429d401 P 2ae2e4cfa8764352a502fbe7ff3009c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:47.687155 21740 ts_tablet_manager.cc:1905] T 97df4001b50a46679e7438e6b429d401 P bac6c1b9f519437f82e706c280a0f13f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:47.688719 21739 ts_tablet_manager.cc:1918] T 97df4001b50a46679e7438e6b429d401 P 2ae2e4cfa8764352a502fbe7ff3009c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:15:47.688802 21739 log.cc:1196] T 97df4001b50a46679e7438e6b429d401 P 2ae2e4cfa8764352a502fbe7ff3009c4: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-1/wal/wals/97df4001b50a46679e7438e6b429d401
I20260929 03:15:47.689005 21739 ts_tablet_manager.cc:1939] T 97df4001b50a46679e7438e6b429d401 P 2ae2e4cfa8764352a502fbe7ff3009c4: Deleting consensus metadata
I20260929 03:15:47.689440 21740 ts_tablet_manager.cc:1918] T 97df4001b50a46679e7438e6b429d401 P bac6c1b9f519437f82e706c280a0f13f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:15:47.689495 21740 log.cc:1196] T 97df4001b50a46679e7438e6b429d401 P bac6c1b9f519437f82e706c280a0f13f: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/97df4001b50a46679e7438e6b429d401
I20260929 03:15:47.689538 21157 catalog_manager.cc:4841] TS 2ae2e4cfa8764352a502fbe7ff3009c4 (127.13.238.130:35993): tablet 97df4001b50a46679e7438e6b429d401 (table test-table [id=52b1d1a3f2ea4ab397ba36c38df31d3d]) successfully deleted
I20260929 03:15:47.689759 21740 ts_tablet_manager.cc:1939] T 97df4001b50a46679e7438e6b429d401 P bac6c1b9f519437f82e706c280a0f13f: Deleting consensus metadata
I20260929 03:15:47.690320 21157 catalog_manager.cc:4841] TS bac6c1b9f519437f82e706c280a0f13f (127.13.238.131:38001): tablet 97df4001b50a46679e7438e6b429d401 (table test-table [id=52b1d1a3f2ea4ab397ba36c38df31d3d]) successfully deleted
I20260929 03:15:47.693596 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 21358
I20260929 03:15:47.693619 21713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:47.693383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:39585 (local address 127.0.0.1:56622)
0929 03:15:47.693487 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:47.693541 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.129:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 03:15:47.693809 21155 catalog_manager.cc:4584] TS 4350ad5a78764d1db39c247d759ce213 (127.13.238.129:39585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 97df4001b50a46679e7438e6b429d401: Network error: Client connection negotiation failed: client connection to 127.13.238.129:39585: connect: Connection refused (error 111)
I20260929 03:15:47.701877 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 21494
I20260929 03:15:47.707226 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 20843
I20260929 03:15:47.711952 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 20952
2026-09-29T03:15:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5243 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:15:47 dist-test-slave-nbjg krb5kdc[21744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:15:47 dist-test-slave-nbjg krb5kdc[21744](info): set up 2 sockets
Sep 29 03:15:47 dist-test-slave-nbjg krb5kdc[21744](info): commencing operation
krb5kdc: starting...
W20260929 03:15:48.662714 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:15:48 dist-test-slave-nbjg krb5kdc[21744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:15:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:15:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.190.keytab.
Entry for principal oryx/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.190.keytab.
I20260929 03:15:48.781248 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:42031
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.190.keytab
--principal=oryx/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:42031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:48.889614 21757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:48.889904 21757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:48.889959 21757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:48.893958 21757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:48.894032 21757 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 03:15:48.894053 21757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:48.894078 21757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:48.894098 21757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:48.894126 21757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:15:48.899518 21757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:42031
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.190.keytab
--principal=oryx/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:42031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:48.900599 21757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:48.901633 21757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:48.909744 21765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:48.909744 21762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:48.909896 21757 server_base.cc:1034] running on GCE node
W20260929 03:15:48.909780 21763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:48.910563 21757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:48.911644 21757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:48.912811 21757 hybrid_clock.cc:648] HybridClock initialized: now 1790651748912791 us; error 32 us; skew 500 ppm
Sep 29 03:15:48 dist-test-slave-nbjg krb5kdc[21744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651748, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:48.915454 21757 init.cc:377] Logged in from keytab as oryx/127.13.238.190@KRBTEST.COM (short username oryx)
I20260929 03:15:48.916296 21757 webserver.cc:460] Webserver started at http://127.13.238.190:34301/ using document root <none> and password file <none>
I20260929 03:15:48.916814 21757 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:48.916869 21757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:48.917037 21757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:48.919072 21757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "0f3c46901f2847f38c9ff7db5f86fe72"
format_stamp: "Formatted at 2026-09-29 03:15:48 on dist-test-slave-nbjg"
server_key: "e4df903d435664673b80aa6d5ddce7da"
server_key_iv: "5204aa0672de6194f118077a35fff2e1"
server_key_version: "encryptionkey@0"
I20260929 03:15:48.919567 21757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "0f3c46901f2847f38c9ff7db5f86fe72"
format_stamp: "Formatted at 2026-09-29 03:15:48 on dist-test-slave-nbjg"
server_key: "e4df903d435664673b80aa6d5ddce7da"
server_key_iv: "5204aa0672de6194f118077a35fff2e1"
server_key_version: "encryptionkey@0"
I20260929 03:15:48.923122 21757 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:15:48.925458 21772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:48.926749 21757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:48.926854 21757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "0f3c46901f2847f38c9ff7db5f86fe72"
format_stamp: "Formatted at 2026-09-29 03:15:48 on dist-test-slave-nbjg"
server_key: "e4df903d435664673b80aa6d5ddce7da"
server_key_iv: "5204aa0672de6194f118077a35fff2e1"
server_key_version: "encryptionkey@0"
I20260929 03:15:48.926956 21757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:48.950320 21757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:48.953604 21757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:48.953809 21757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:48.961527 21757 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:42031
I20260929 03:15:48.961540 21834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:42031 every 8 connection(s)
I20260929 03:15:48.962519 21757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:15:48.965363 21835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:48.967151 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 21757
I20260929 03:15:48.967264 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:15:48.967584 14266 external_mini_cluster.cc:1428] Setting key cef5ba17697c4e4d11aa804777f6cdf0
I20260929 03:15:48.971774 21835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72: Bootstrap starting.
I20260929 03:15:48.974067 21835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:48.974813 21835 log.cc:824] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72: Log is configured to *not* fsync() on all Append() calls
Sep 29 03:15:48 dist-test-slave-nbjg krb5kdc[21744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM
I20260929 03:15:48.977114 21835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72: No bootstrap required, opened a new log
I20260929 03:15:48.981009 21835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3c46901f2847f38c9ff7db5f86fe72" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42031 } }
I20260929 03:15:48.981231 21835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:48.981266 21835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3c46901f2847f38c9ff7db5f86fe72, State: Initialized, Role: FOLLOWER
I20260929 03:15:48.981315 21838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:48.969031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59464 (local address 127.13.238.190:42031)
0929 03:15:48.969472 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:48.969487 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:48.969522 (+ 35us) server_negotiation.cc:407] Connection header received
0929 03:15:48.970152 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:48.970169 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:48.970593 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:48.970906 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:48.971686 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:48.972699 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:48.973528 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:48.973993 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:48.974135 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:48.976794 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:48.976800 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:48.976814 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:48.976840 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:48.978456 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:48.978913 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:48.978917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:48.978921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:48.979028 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:48.979310 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:48.979313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:48.979314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:48.979733 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:48.979913 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:48.980208 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 03:15:48.980491 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":141,"threads_started":1}
I20260929 03:15:48.981740 21835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3c46901f2847f38c9ff7db5f86fe72" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42031 } }
I20260929 03:15:48.981886 21835 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:15:48.981950 21835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:15:48.982048 21835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:48.982950 21835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3c46901f2847f38c9ff7db5f86fe72" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42031 } }
I20260929 03:15:48.983274 21835 leader_election.cc:302] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f3c46901f2847f38c9ff7db5f86fe72; no voters:
I20260929 03:15:48.983609 21835 leader_election.cc:288] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:15:48.983700 21840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:48.983891 21840 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72 [term 1 LEADER]: Becoming Leader. State: Replica: 0f3c46901f2847f38c9ff7db5f86fe72, State: Running, Role: LEADER
I20260929 03:15:48.984189 21840 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3c46901f2847f38c9ff7db5f86fe72" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42031 } }
I20260929 03:15:48.984800 21835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:48.985809 21842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f3c46901f2847f38c9ff7db5f86fe72. Latest consensus state: current_term: 1 leader_uuid: "0f3c46901f2847f38c9ff7db5f86fe72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3c46901f2847f38c9ff7db5f86fe72" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42031 } } }
I20260929 03:15:48.985839 21841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f3c46901f2847f38c9ff7db5f86fe72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3c46901f2847f38c9ff7db5f86fe72" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42031 } } }
I20260929 03:15:48.985951 21842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:48.985952 21841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72 [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:48.986336 21849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:15:48.989323 21849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:15:48.995150 21849 catalog_manager.cc:1347] Generated new cluster ID: b652b0a8b2e24c68b7b26c6b53dd15bd
I20260929 03:15:48.995232 21849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:15:49.019025 21849 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:15:49.020437 21849 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:15:49.026971 21849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72: Generated new TSK 0
I20260929 03:15:49.027832 21849 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.13.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.129.keytab.
Entry for principal oryx/127.13.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.129.keytab.
I20260929 03:15:49.057652 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:0
--local_ip_for_outbound_sockets=127.13.238.129
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:42031
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.129.keytab
--principal=oryx/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:49.165745 21861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:49.166008 21861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:49.166069 21861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:49.170156 21861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:49.170248 21861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:49.170303 21861 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:15:49.170358 21861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:15:49.175738 21861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.129.keytab
--principal=oryx/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.238.190:42031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:49.176842 21861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:49.177932 21861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:49.187620 21869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:49.187687 21867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:49.187898 21861 server_base.cc:1034] running on GCE node
W20260929 03:15:49.187687 21866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:49.188285 21861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:49.188941 21861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:49.190111 21861 hybrid_clock.cc:648] HybridClock initialized: now 1790651749190089 us; error 36 us; skew 500 ppm
Sep 29 03:15:49 dist-test-slave-nbjg krb5kdc[21744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651749, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:49.192883 21861 init.cc:377] Logged in from keytab as oryx/127.13.238.129@KRBTEST.COM (short username oryx)
I20260929 03:15:49.193789 21861 webserver.cc:460] Webserver started at http://127.13.238.129:43263/ using document root <none> and password file <none>
I20260929 03:15:49.194365 21861 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:49.194427 21861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:49.194626 21861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:49.196651 21861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-0/data/instance:
uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a"
format_stamp: "Formatted at 2026-09-29 03:15:49 on dist-test-slave-nbjg"
server_key: "cf145466dbabbcee306876f8f9c42a9c"
server_key_iv: "c020eb3b98cf5df77e62743e391e86cf"
server_key_version: "encryptionkey@0"
I20260929 03:15:49.197196 21861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance:
uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a"
format_stamp: "Formatted at 2026-09-29 03:15:49 on dist-test-slave-nbjg"
server_key: "cf145466dbabbcee306876f8f9c42a9c"
server_key_iv: "c020eb3b98cf5df77e62743e391e86cf"
server_key_version: "encryptionkey@0"
I20260929 03:15:49.201298 21861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260929 03:15:49.204041 21876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:49.205629 21861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:15:49.205742 21861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a"
format_stamp: "Formatted at 2026-09-29 03:15:49 on dist-test-slave-nbjg"
server_key: "cf145466dbabbcee306876f8f9c42a9c"
server_key_iv: "c020eb3b98cf5df77e62743e391e86cf"
server_key_version: "encryptionkey@0"
I20260929 03:15:49.205863 21861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:49.217142 21861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:49.220250 21861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:49.220424 21861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:49.222090 21861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:49.222139 21861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:49.222210 21861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:49.222237 21861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:15:49 dist-test-slave-nbjg krb5kdc[21744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651749, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM
I20260929 03:15:49.231834 21861 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:36693
I20260929 03:15:49.231874 21991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:36693 every 8 connection(s)
I20260929 03:15:49.232874 21861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:15:49.234097 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 21861
I20260929 03:15:49.234249 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance
I20260929 03:15:49.234579 14266 external_mini_cluster.cc:1428] Setting key e53e7e4cf18196c41a425cd2d3ee00b6
I20260929 03:15:49.236202 21838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:49.222929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:44325 (local address 127.13.238.190:42031)
0929 03:15:49.223080 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:49.223085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:49.223960 (+ 875us) server_negotiation.cc:407] Connection header received
0929 03:15:49.225066 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:49.225070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:49.225132 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:49.225197 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:49.226802 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.227406 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.228898 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.229239 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.229400 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:49.232289 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:49.232292 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:49.232295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:49.232322 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:49.233938 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:49.234533 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:49.234537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:49.234539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:49.234640 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:49.235071 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:49.235075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:49.235077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:49.235319 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:49.235417 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:49.235944 (+ 527us) server_negotiation.cc:299] Negotiation successful
0929 03:15:49.236082 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:15:49.237222 21885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:49.223328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42031 (local address 127.13.238.129:44325)
0929 03:15:49.223801 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:49.223852 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:49.224815 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:49.225604 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:49.225611 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:49.226017 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:49.226652 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:49.226668 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.227523 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.227526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:49.228771 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:49.228782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.229432 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.229435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:49.229572 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:49.230271 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:49.230292 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:49.232111 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:49.234061 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:49.234067 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:49.234082 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:49.234428 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:49.234768 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:49.234771 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:49.234773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:49.234952 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:49.235421 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:49.235427 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:49.235672 (+ 245us) client_negotiation.cc:769] Sending connection context
0929 03:15:49.235862 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 03:15:49.236194 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":105,"threads_started":1}
I20260929 03:15:49.238866 21992 heartbeater.cc:347] Connected to a master server at 127.13.238.190:42031
I20260929 03:15:49.239008 21992 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:49.239598 21992 heartbeater.cc:511] Master 127.13.238.190:42031 requested a full tablet report, sending...
I20260929 03:15:49.241240 21788 ts_manager.cc:194] Registered new tserver with Master: a33a5ddf88af461f9ae2e48f79a8ff4a (127.13.238.129:36693)
I20260929 03:15:49.242573 21788 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.238.129@KRBTEST.COM'} at 127.13.238.129:44325
WARNING: no policy specified for oryx/127.13.238.130@KRBTEST.COM; defaulting to no policy
I20260929 03:15:49.249226 21838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:49.242316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:52615 (local address 127.13.238.190:42031)
0929 03:15:49.242421 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:49.242425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:49.242438 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:15:49.242472 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:49.242474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:49.242512 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:49.242585 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:49.243094 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.243831 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.244740 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.245057 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.245159 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:49.246407 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:49.246410 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:49.246412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:49.246433 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:49.247720 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:49.248121 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:49.248125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:49.248126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:49.248198 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:49.248444 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:49.248447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:49.248449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:49.248644 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:49.248725 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:49.248974 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 03:15:49.249075 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:15:49.249243 21999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:49.241204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42031 (local address 127.13.238.129:52615)
0929 03:15:49.241507 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:49.241522 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:49.241604 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:49.242630 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:49.242634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:49.242857 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:49.242974 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:49.242982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.243969 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.243973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:49.244611 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:49.244623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.245176 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.245180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:49.245279 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:49.245808 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:49.245822 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:49.246285 (+ 463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:49.247832 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:49.247835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:49.247837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:49.248032 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:49.248277 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:49.248280 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:49.248282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:49.248367 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:49.248730 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:49.248734 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:49.248830 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 03:15:49.248914 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 03:15:49.249057 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.238.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.130.keytab.
Entry for principal oryx/127.13.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.130.keytab.
I20260929 03:15:49.265925 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:0
--local_ip_for_outbound_sockets=127.13.238.130
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:42031
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.130.keytab
--principal=oryx/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:49.374377 22001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:49.374645 22001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:49.374687 22001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:49.378784 22001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:49.378863 22001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:49.378911 22001 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:15:49.378965 22001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:15:49.384367 22001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.130.keytab
--principal=oryx/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.238.190:42031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:49.385545 22001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:49.386682 22001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:49.396117 22007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:49.396229 22009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:49.396117 22006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:49.396317 22001 server_base.cc:1034] running on GCE node
I20260929 03:15:49.396808 22001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:49.397410 22001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:49.398558 22001 hybrid_clock.cc:648] HybridClock initialized: now 1790651749398531 us; error 36 us; skew 500 ppm
Sep 29 03:15:49 dist-test-slave-nbjg krb5kdc[21744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651749, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:49.401068 22001 init.cc:377] Logged in from keytab as oryx/127.13.238.130@KRBTEST.COM (short username oryx)
I20260929 03:15:49.401926 22001 webserver.cc:460] Webserver started at http://127.13.238.130:42507/ using document root <none> and password file <none>
I20260929 03:15:49.402498 22001 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:49.402565 22001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:49.402767 22001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:49.404748 22001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-1/data/instance:
uuid: "971840703d2d45d08e9f2727920036c6"
format_stamp: "Formatted at 2026-09-29 03:15:49 on dist-test-slave-nbjg"
server_key: "351ec57443bbf2fdeb8b7fc233760799"
server_key_iv: "58ec0155698bedce0481601b50abf89c"
server_key_version: "encryptionkey@0"
I20260929 03:15:49.405272 22001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance:
uuid: "971840703d2d45d08e9f2727920036c6"
format_stamp: "Formatted at 2026-09-29 03:15:49 on dist-test-slave-nbjg"
server_key: "351ec57443bbf2fdeb8b7fc233760799"
server_key_iv: "58ec0155698bedce0481601b50abf89c"
server_key_version: "encryptionkey@0"
I20260929 03:15:49.408854 22001 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 03:15:49.411302 22016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:49.412519 22001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:49.412624 22001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "971840703d2d45d08e9f2727920036c6"
format_stamp: "Formatted at 2026-09-29 03:15:49 on dist-test-slave-nbjg"
server_key: "351ec57443bbf2fdeb8b7fc233760799"
server_key_iv: "58ec0155698bedce0481601b50abf89c"
server_key_version: "encryptionkey@0"
I20260929 03:15:49.412740 22001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:49.423545 22001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:49.426306 22001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:49.426476 22001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:49.427937 22001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:49.427987 22001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:49.428040 22001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:49.428068 22001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:15:49 dist-test-slave-nbjg krb5kdc[21744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651749, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM
I20260929 03:15:49.438486 22001 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:43511
I20260929 03:15:49.438524 22131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:43511 every 8 connection(s)
I20260929 03:15:49.439632 22001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:15:49.441756 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 22001
I20260929 03:15:49.441784 21838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:49.428737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:46991 (local address 127.13.238.190:42031)
0929 03:15:49.428878 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:49.428882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:49.429995 (+ 1113us) server_negotiation.cc:407] Connection header received
0929 03:15:49.431175 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:49.431180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:49.431245 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:49.431316 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:49.432844 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.433523 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.434890 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.435121 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.435215 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:49.437971 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:49.437974 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:49.437977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:49.438006 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:49.439752 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:49.440366 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:49.440369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:49.440371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:49.440474 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:49.440784 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:49.440787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:49.440789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:49.441010 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:49.441110 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:49.441554 (+ 444us) server_negotiation.cc:299] Negotiation successful
0929 03:15:49.441658 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:15:49.441886 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance
I20260929 03:15:49.442183 14266 external_mini_cluster.cc:1428] Setting key 1f34ef5e6991d8d7c1a155e8195c2db3
I20260929 03:15:49.442963 22025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:49.429323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42031 (local address 127.13.238.130:46991)
0929 03:15:49.429826 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:49.429901 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:49.430918 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:49.431741 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:49.431750 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:49.432101 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:49.432691 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:49.432706 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.433642 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.433646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:49.434761 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:49.434773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.435285 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.435300 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:49.435470 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:49.436084 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:49.436100 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:49.437730 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:49.439885 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:49.439892 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:49.439908 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:49.440214 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:49.440557 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:49.440560 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:49.440562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:49.440695 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:49.441113 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:49.441119 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:49.441352 (+ 233us) client_negotiation.cc:769] Sending connection context
0929 03:15:49.441528 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 03:15:49.441926 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":136,"threads_started":1}
I20260929 03:15:49.444239 22132 heartbeater.cc:347] Connected to a master server at 127.13.238.190:42031
I20260929 03:15:49.444393 22132 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:49.444957 22132 heartbeater.cc:511] Master 127.13.238.190:42031 requested a full tablet report, sending...
I20260929 03:15:49.445976 21788 ts_manager.cc:194] Registered new tserver with Master: 971840703d2d45d08e9f2727920036c6 (127.13.238.130:43511)
I20260929 03:15:49.446663 21788 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.238.130@KRBTEST.COM'} at 127.13.238.130:46991
WARNING: no policy specified for oryx/127.13.238.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.238.131@KRBTEST.COM" created.
I20260929 03:15:49.457273 21838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:49.446340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:54875 (local address 127.13.238.190:42031)
0929 03:15:49.446453 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:49.446457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:49.450503 (+ 4046us) server_negotiation.cc:407] Connection header received
0929 03:15:49.450637 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:49.450639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:49.450681 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:49.450751 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:49.451289 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.451814 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.452656 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.452890 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.452968 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:49.454064 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:49.454067 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:49.454069 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:49.454090 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:49.455641 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:49.456113 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:49.456115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:49.456117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:49.456192 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:49.456451 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:49.456454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:49.456455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:49.456653 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:49.456762 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:49.457006 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 03:15:49.457112 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:15:49.457301 22139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:49.446287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42031 (local address 127.13.238.130:54875)
0929 03:15:49.450409 (+ 4122us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:49.450424 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:49.450536 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:49.450787 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:49.450791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:49.450991 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:49.451111 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:49.451132 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.451933 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.451936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:49.452559 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:49.452568 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.452984 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.452986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:49.453065 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:49.453534 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:49.453545 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:49.453942 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:49.455765 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:49.455768 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:49.455770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:49.456011 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:49.456275 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:49.456278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:49.456280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:49.456373 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:49.456752 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:49.456755 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:49.456852 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 03:15:49.456942 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 03:15:49.457093 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4025,"thread_start_us":86,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.131.keytab.
Entry for principal oryx/127.13.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.131.keytab.
I20260929 03:15:49.470443 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:0
--local_ip_for_outbound_sockets=127.13.238.131
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:42031
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.131.keytab
--principal=oryx/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:49.580005 22141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:49.580257 22141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:49.580298 22141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:49.584203 22141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:49.584262 22141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:49.584316 22141 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:15:49.584367 22141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:15:49.589764 22141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.131.keytab
--principal=oryx/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.238.190:42031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:49.590927 22141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:49.591945 22141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:49.601747 22147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:49.601948 22146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:49.601956 22141 server_base.cc:1034] running on GCE node
W20260929 03:15:49.601969 22149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:49.602548 22141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:49.603217 22141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:49.604382 22141 hybrid_clock.cc:648] HybridClock initialized: now 1790651749604359 us; error 30 us; skew 500 ppm
Sep 29 03:15:49 dist-test-slave-nbjg krb5kdc[21744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651749, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:49.607077 22141 init.cc:377] Logged in from keytab as oryx/127.13.238.131@KRBTEST.COM (short username oryx)
I20260929 03:15:49.607946 22141 webserver.cc:460] Webserver started at http://127.13.238.131:43139/ using document root <none> and password file <none>
I20260929 03:15:49.608497 22141 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:49.608560 22141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:49.608755 22141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:49.610914 22141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-2/data/instance:
uuid: "5f130cc04a74491791f8c2b9b5a38f4d"
format_stamp: "Formatted at 2026-09-29 03:15:49 on dist-test-slave-nbjg"
server_key: "f89dabcc200df8caebad1e2e6fe8a117"
server_key_iv: "e633c501eefae37302eebf92947dc66c"
server_key_version: "encryptionkey@0"
I20260929 03:15:49.611441 22141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance:
uuid: "5f130cc04a74491791f8c2b9b5a38f4d"
format_stamp: "Formatted at 2026-09-29 03:15:49 on dist-test-slave-nbjg"
server_key: "f89dabcc200df8caebad1e2e6fe8a117"
server_key_iv: "e633c501eefae37302eebf92947dc66c"
server_key_version: "encryptionkey@0"
I20260929 03:15:49.615018 22141 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:15:49.617332 22156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:49.618562 22141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:49.618676 22141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "5f130cc04a74491791f8c2b9b5a38f4d"
format_stamp: "Formatted at 2026-09-29 03:15:49 on dist-test-slave-nbjg"
server_key: "f89dabcc200df8caebad1e2e6fe8a117"
server_key_iv: "e633c501eefae37302eebf92947dc66c"
server_key_version: "encryptionkey@0"
I20260929 03:15:49.618767 22141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:49.630667 22141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:49.633689 22141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:49.633862 22141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:49.635326 22141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:49.635375 22141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:49.635429 22141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:49.635458 22141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:15:49 dist-test-slave-nbjg krb5kdc[21744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651749, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM
I20260929 03:15:49.646010 22141 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:36921
I20260929 03:15:49.646430 22271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:36921 every 8 connection(s)
I20260929 03:15:49.647048 22141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:15:49.649103 21838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:49.636137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:48617 (local address 127.13.238.190:42031)
0929 03:15:49.636258 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:49.636262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:49.637348 (+ 1086us) server_negotiation.cc:407] Connection header received
0929 03:15:49.638412 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:49.638416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:49.638479 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:49.638538 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:49.639975 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.640792 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.642168 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.642550 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.642669 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:49.645378 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:49.645382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:49.645385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:49.645417 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:49.647031 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:49.647540 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:49.647544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:49.647546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:49.647644 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:49.647985 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:49.647988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:49.647990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:49.648219 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:49.648314 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:49.648816 (+ 502us) server_negotiation.cc:299] Negotiation successful
0929 03:15:49.648939 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:15:49.649892 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:49.636685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42031 (local address 127.13.238.131:48617)
0929 03:15:49.637186 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:49.637227 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:49.638138 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:49.638960 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:49.638967 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:49.639280 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:49.639812 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:49.639830 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.640934 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.640938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:49.642034 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:49.642047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.642728 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.642733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:49.642928 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:49.643576 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:49.643593 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:49.645188 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:49.647152 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:49.647158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:49.647175 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:49.647439 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:49.647750 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:49.647753 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:49.647755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:49.647892 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:49.648332 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:49.648339 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:49.648578 (+ 239us) client_negotiation.cc:769] Sending connection context
0929 03:15:49.648746 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 03:15:49.649074 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":131,"threads_started":1}
I20260929 03:15:49.651063 22272 heartbeater.cc:347] Connected to a master server at 127.13.238.190:42031
I20260929 03:15:49.651201 22272 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:49.651681 22272 heartbeater.cc:511] Master 127.13.238.190:42031 requested a full tablet report, sending...
I20260929 03:15:49.652704 21789 ts_manager.cc:194] Registered new tserver with Master: 5f130cc04a74491791f8c2b9b5a38f4d (127.13.238.131:36921)
I20260929 03:15:49.653414 21789 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.238.131@KRBTEST.COM'} at 127.13.238.131:48617
I20260929 03:15:49.656565 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 22141
I20260929 03:15:49.656679 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance
I20260929 03:15:49.656960 14266 external_mini_cluster.cc:1428] Setting key d2b781e60a27d2e0c187340445c28b3d
I20260929 03:15:49.659013 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:15:49.660085 21838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:49.653166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:44627 (local address 127.13.238.190:42031)
0929 03:15:49.653280 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:49.653284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:49.653436 (+ 152us) server_negotiation.cc:407] Connection header received
0929 03:15:49.653546 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:49.653549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:49.653587 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:49.653656 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:49.654124 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.654878 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.655664 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.655874 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.655951 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:49.657037 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:49.657041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:49.657043 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:49.657067 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:49.658570 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:49.659040 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:49.659043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:49.659045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:49.659130 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:49.659366 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:49.659369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:49.659370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:49.659531 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:49.659612 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:49.659847 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 03:15:49.659943 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:15:49.660097 22278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:49.653077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42031 (local address 127.13.238.131:44627)
0929 03:15:49.653365 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:49.653376 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:49.653456 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:49.653694 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:49.653697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:49.653907 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:49.654018 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:49.654024 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.655009 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.655012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:49.655562 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:49.655570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.655993 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.655996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:49.656071 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:49.656569 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:49.656581 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:49.656911 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:49.658685 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:49.658689 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:49.658690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:49.658957 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:49.659200 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:49.659203 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:49.659205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:49.659291 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:49.659614 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:49.659618 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:49.659702 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 03:15:49.659785 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 03:15:49.659923 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":90,"threads_started":1}
Sep 29 03:15:49 dist-test-slave-nbjg krb5kdc[21744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651749, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:15:49 dist-test-slave-nbjg krb5kdc[21744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:15:49 dist-test-slave-nbjg krb5kdc[21744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:15:49.681898 21838 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:49.675833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59480 (local address 127.13.238.190:42031)
0929 03:15:49.675973 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:49.675977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:49.675990 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:15:49.676032 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:49.676034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:49.676074 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:49.676147 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:49.676880 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.677418 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.678306 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.678505 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.678579 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:49.681760 (+ 3181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59480: BlockingRecv error: recv got EOF from 127.0.0.1:59480 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 03:15:49 dist-test-slave-nbjg krb5kdc[21744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651749, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM
I20260929 03:15:49.690412 21838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:49.683246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59484 (local address 127.13.238.190:42031)
0929 03:15:49.683358 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:49.683362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:49.683429 (+ 67us) server_negotiation.cc:407] Connection header received
0929 03:15:49.683562 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:49.683565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:49.683623 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:49.683696 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:49.684209 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.684683 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.685447 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.685651 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.685738 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:49.687931 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:49.687935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:49.687937 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:49.687961 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:49.689022 (+ 1061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:49.689401 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:49.689403 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:49.689404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:49.689466 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:49.689731 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:49.689734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:49.689735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:49.689904 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:49.690003 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:49.690159 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 03:15:49.690291 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:15:49.693064 21789 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:15:49.696130 21789 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:15:49.709767 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:49.704337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:36921 (local address 127.0.0.1:60832)
0929 03:15:49.705143 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:49.705193 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:49.705329 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:49.705954 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:49.705960 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:49.705981 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:49.706361 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:49.706368 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.707314 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.707317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:49.708456 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:49.708465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.709231 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.709235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:49.709352 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:49.709399 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 03:15:49.709505 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 03:15:49.709620 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":93,"threads_started":1}
I20260929 03:15:49.709782 22298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:49.704839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.13.238.131:36921)
0929 03:15:49.705457 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:49.705471 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:49.705494 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:15:49.705574 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:49.705581 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:49.705815 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:49.705954 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:49.706462 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.707178 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.708568 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.709124 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.709266 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:49.709403 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:49.709542 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 03:15:49.709634 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":102,"threads_started":1}
I20260929 03:15:49.710757 22294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:49.704548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:36693 (local address 127.0.0.1:41114)
0929 03:15:49.705572 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:49.705584 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:49.705666 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:49.706243 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:49.706246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:49.706263 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:49.706370 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:49.706376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.707699 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.707702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:49.708998 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:49.709007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.710284 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.710287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:49.710460 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:49.710486 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 03:15:49.710558 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 03:15:49.710631 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"mutex_wait_us":104,"thread_start_us":91,"threads_started":1}
I20260929 03:15:49.711272 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:49.704198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:43511 (local address 127.0.0.1:33422)
0929 03:15:49.705542 (+ 1344us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:49.705555 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:49.705646 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:49.706098 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:49.706101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:49.706115 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:49.706239 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:49.706247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.707699 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.707702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:49.708996 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:49.709007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.710688 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.710691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:49.710791 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:49.711000 (+ 209us) client_negotiation.cc:769] Sending connection context
0929 03:15:49.711077 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 03:15:49.711148 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":39,"thread_start_us":90,"threads_started":1}
I20260929 03:15:49.711354 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:49.704363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33422 (local address 127.13.238.130:43511)
0929 03:15:49.705263 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:49.705275 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:49.705633 (+ 358us) server_negotiation.cc:407] Connection header received
0929 03:15:49.705747 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:49.705752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:49.705946 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:49.706093 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:49.706354 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.707549 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.709124 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.710338 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.710509 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:49.710626 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:49.711124 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 03:15:49.711215 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":801,"thread_start_us":83,"threads_started":1}
I20260929 03:15:49.712932 22206 tablet_service.cc:1467] Processing CreateTablet for tablet b91e1944461f4b39b1107a4c61542211 (DEFAULT_TABLE table=test-table [id=71b47faee93541ccb526f1817601e4d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:49.713553 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:49.705017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41114 (local address 127.13.238.129:36693)
0929 03:15:49.705433 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:49.705446 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:49.705749 (+ 303us) server_negotiation.cc:407] Connection header received
0929 03:15:49.705826 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:49.705830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:49.705968 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:49.706092 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:49.706489 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.707579 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.709147 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.709944 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.710066 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:49.710185 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:49.710598 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 03:15:49.713414 (+ 2816us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":81,"threads_started":1}
I20260929 03:15:49.713936 22206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91e1944461f4b39b1107a4c61542211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:49.713980 22066 tablet_service.cc:1467] Processing CreateTablet for tablet b91e1944461f4b39b1107a4c61542211 (DEFAULT_TABLE table=test-table [id=71b47faee93541ccb526f1817601e4d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:49.715046 22066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91e1944461f4b39b1107a4c61542211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:49.716162 21926 tablet_service.cc:1467] Processing CreateTablet for tablet b91e1944461f4b39b1107a4c61542211 (DEFAULT_TABLE table=test-table [id=71b47faee93541ccb526f1817601e4d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:49.717145 21926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91e1944461f4b39b1107a4c61542211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:49.721539 22300 tablet_bootstrap.cc:492] T b91e1944461f4b39b1107a4c61542211 P 971840703d2d45d08e9f2727920036c6: Bootstrap starting.
I20260929 03:15:49.722860 22301 tablet_bootstrap.cc:492] T b91e1944461f4b39b1107a4c61542211 P 5f130cc04a74491791f8c2b9b5a38f4d: Bootstrap starting.
I20260929 03:15:49.723687 22300 tablet_bootstrap.cc:654] T b91e1944461f4b39b1107a4c61542211 P 971840703d2d45d08e9f2727920036c6: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:49.724386 22300 log.cc:824] T b91e1944461f4b39b1107a4c61542211 P 971840703d2d45d08e9f2727920036c6: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:49.724778 22302 tablet_bootstrap.cc:492] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a: Bootstrap starting.
I20260929 03:15:49.725109 22301 tablet_bootstrap.cc:654] T b91e1944461f4b39b1107a4c61542211 P 5f130cc04a74491791f8c2b9b5a38f4d: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:49.725919 22301 log.cc:824] T b91e1944461f4b39b1107a4c61542211 P 5f130cc04a74491791f8c2b9b5a38f4d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:49.726972 22300 tablet_bootstrap.cc:492] T b91e1944461f4b39b1107a4c61542211 P 971840703d2d45d08e9f2727920036c6: No bootstrap required, opened a new log
I20260929 03:15:49.727005 22302 tablet_bootstrap.cc:654] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:49.727182 22300 ts_tablet_manager.cc:1397] T b91e1944461f4b39b1107a4c61542211 P 971840703d2d45d08e9f2727920036c6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 03:15:49.727566 22301 tablet_bootstrap.cc:492] T b91e1944461f4b39b1107a4c61542211 P 5f130cc04a74491791f8c2b9b5a38f4d: No bootstrap required, opened a new log
I20260929 03:15:49.727738 22301 ts_tablet_manager.cc:1397] T b91e1944461f4b39b1107a4c61542211 P 5f130cc04a74491791f8c2b9b5a38f4d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260929 03:15:49.727752 22302 log.cc:824] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:49.729563 22302 tablet_bootstrap.cc:492] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a: No bootstrap required, opened a new log
I20260929 03:15:49.729749 22302 ts_tablet_manager.cc:1397] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 03:15:49.730854 22301 raft_consensus.cc:348] T b91e1944461f4b39b1107a4c61542211 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } } peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } }
I20260929 03:15:49.730854 22300 raft_consensus.cc:348] T b91e1944461f4b39b1107a4c61542211 P 971840703d2d45d08e9f2727920036c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } } peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } }
I20260929 03:15:49.731081 22300 raft_consensus.cc:374] T b91e1944461f4b39b1107a4c61542211 P 971840703d2d45d08e9f2727920036c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:49.731078 22301 raft_consensus.cc:374] T b91e1944461f4b39b1107a4c61542211 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:49.731139 22301 raft_consensus.cc:729] T b91e1944461f4b39b1107a4c61542211 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f130cc04a74491791f8c2b9b5a38f4d, State: Initialized, Role: FOLLOWER
I20260929 03:15:49.731132 22300 raft_consensus.cc:729] T b91e1944461f4b39b1107a4c61542211 P 971840703d2d45d08e9f2727920036c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971840703d2d45d08e9f2727920036c6, State: Initialized, Role: FOLLOWER
I20260929 03:15:49.731570 22301 consensus_queue.cc:260] T b91e1944461f4b39b1107a4c61542211 P 5f130cc04a74491791f8c2b9b5a38f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } } peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } }
I20260929 03:15:49.731565 22300 consensus_queue.cc:260] T b91e1944461f4b39b1107a4c61542211 P 971840703d2d45d08e9f2727920036c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } } peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } }
I20260929 03:15:49.732491 22300 ts_tablet_manager.cc:1428] T b91e1944461f4b39b1107a4c61542211 P 971840703d2d45d08e9f2727920036c6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 03:15:49.732497 22301 ts_tablet_manager.cc:1428] T b91e1944461f4b39b1107a4c61542211 P 5f130cc04a74491791f8c2b9b5a38f4d: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260929 03:15:49.732827 22132 heartbeater.cc:503] Master 127.13.238.190:42031 was elected leader, sending a full tablet report...
I20260929 03:15:49.732949 22272 heartbeater.cc:503] Master 127.13.238.190:42031 was elected leader, sending a full tablet report...
I20260929 03:15:49.734169 22302 raft_consensus.cc:348] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } } peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } }
I20260929 03:15:49.734429 22302 raft_consensus.cc:374] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:49.734478 22302 raft_consensus.cc:729] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a33a5ddf88af461f9ae2e48f79a8ff4a, State: Initialized, Role: FOLLOWER
I20260929 03:15:49.734866 22302 consensus_queue.cc:260] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } } peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } }
I20260929 03:15:49.735428 21992 heartbeater.cc:503] Master 127.13.238.190:42031 was elected leader, sending a full tablet report...
I20260929 03:15:49.735749 22302 ts_tablet_manager.cc:1428] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 03:15:49.745191 22308 raft_consensus.cc:482] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:15:49.745337 22308 raft_consensus.cc:504] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } } peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } }
I20260929 03:15:49.746356 22308 leader_election.cc:288] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 971840703d2d45d08e9f2727920036c6 (127.13.238.130:43511), 5f130cc04a74491791f8c2b9b5a38f4d (127.13.238.131:36921)
I20260929 03:15:49.751237 22310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:49.746634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:36921 (local address 127.13.238.129:47933)
0929 03:15:49.746920 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:49.746931 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:49.747012 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:49.747213 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:49.747216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:49.747264 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:49.747391 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:49.747396 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.748390 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.748394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:49.749801 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:49.749813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.750740 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.750744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:49.750850 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:49.750926 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 03:15:49.750998 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 03:15:49.751076 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":158,"spinlock_wait_cycles":30464,"thread_start_us":98,"threads_started":1}
I20260929 03:15:49.751228 22298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:49.746682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:47933 (local address 127.13.238.131:36921)
0929 03:15:49.746814 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:49.746818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:49.746994 (+ 176us) server_negotiation.cc:407] Connection header received
0929 03:15:49.747084 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:49.747086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:49.747124 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:49.747197 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:49.747485 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.748271 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.749926 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.750564 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.750684 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:49.750749 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:49.751019 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 03:15:49.751107 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:15:49.751243 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:49.746696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:41685 (local address 127.13.238.130:43511)
0929 03:15:49.746813 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:49.746817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:49.746923 (+ 106us) server_negotiation.cc:407] Connection header received
0929 03:15:49.747069 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:49.747072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:49.747119 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:49.747197 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:49.747487 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.748271 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.749926 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.750584 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.750684 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:49.750736 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:49.751046 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 03:15:49.751126 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:15:49.751641 22309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:49.746523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:43511 (local address 127.13.238.129:41685)
0929 03:15:49.746832 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:49.746847 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:49.746950 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:49.747213 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:49.747216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:49.747234 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:49.747379 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:49.747387 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.748390 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.748394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:49.749801 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:49.749813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.750798 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.750802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:49.750908 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:49.750931 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:15:49.751428 (+ 497us) client_negotiation.cc:240] Negotiation successful
0929 03:15:49.751503 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":44,"thread_start_us":84,"threads_started":1}
I20260929 03:15:49.751847 22226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e1944461f4b39b1107a4c61542211" candidate_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" is_pre_election: true
I20260929 03:15:49.752164 22226 raft_consensus.cc:2454] T b91e1944461f4b39b1107a4c61542211 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a33a5ddf88af461f9ae2e48f79a8ff4a in term 0.
I20260929 03:15:49.752241 22086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e1944461f4b39b1107a4c61542211" candidate_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971840703d2d45d08e9f2727920036c6" is_pre_election: true
I20260929 03:15:49.752524 22086 raft_consensus.cc:2454] T b91e1944461f4b39b1107a4c61542211 P 971840703d2d45d08e9f2727920036c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a33a5ddf88af461f9ae2e48f79a8ff4a in term 0.
I20260929 03:15:49.752959 21880 leader_election.cc:302] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 971840703d2d45d08e9f2727920036c6, a33a5ddf88af461f9ae2e48f79a8ff4a; no voters:
I20260929 03:15:49.753224 22308 raft_consensus.cc:2781] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:15:49.753312 22308 raft_consensus.cc:482] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:15:49.753363 22308 raft_consensus.cc:3037] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:49.754388 22308 raft_consensus.cc:504] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } } peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } }
I20260929 03:15:49.754705 22308 leader_election.cc:288] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a [CANDIDATE]: Term 1 election: Requested vote from peers 971840703d2d45d08e9f2727920036c6 (127.13.238.130:43511), 5f130cc04a74491791f8c2b9b5a38f4d (127.13.238.131:36921)
I20260929 03:15:49.755033 22086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e1944461f4b39b1107a4c61542211" candidate_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971840703d2d45d08e9f2727920036c6"
I20260929 03:15:49.755061 22226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e1944461f4b39b1107a4c61542211" candidate_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f130cc04a74491791f8c2b9b5a38f4d"
I20260929 03:15:49.755180 22226 raft_consensus.cc:3037] T b91e1944461f4b39b1107a4c61542211 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:49.755167 22086 raft_consensus.cc:3037] T b91e1944461f4b39b1107a4c61542211 P 971840703d2d45d08e9f2727920036c6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:49.756521 22226 raft_consensus.cc:2454] T b91e1944461f4b39b1107a4c61542211 P 5f130cc04a74491791f8c2b9b5a38f4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a33a5ddf88af461f9ae2e48f79a8ff4a in term 1.
I20260929 03:15:49.756830 21877 leader_election.cc:302] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f130cc04a74491791f8c2b9b5a38f4d, a33a5ddf88af461f9ae2e48f79a8ff4a; no voters:
I20260929 03:15:49.756843 22086 raft_consensus.cc:2454] T b91e1944461f4b39b1107a4c61542211 P 971840703d2d45d08e9f2727920036c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a33a5ddf88af461f9ae2e48f79a8ff4a in term 1.
I20260929 03:15:49.757015 22308 raft_consensus.cc:2781] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:49.757210 22308 raft_consensus.cc:686] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 1 LEADER]: Becoming Leader. State: Replica: a33a5ddf88af461f9ae2e48f79a8ff4a, State: Running, Role: LEADER
I20260929 03:15:49.757522 22308 consensus_queue.cc:237] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } } peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } }
I20260929 03:15:49.760756 21787 catalog_manager.cc:5495] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a reported cstate change: term changed from 0 to 1, leader changed from <none> to a33a5ddf88af461f9ae2e48f79a8ff4a (127.13.238.129). New cstate: current_term: 1 leader_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } health_report { overall_health: HEALTHY } } }
I20260929 03:15:49.770973 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:49.767939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41118 (local address 127.13.238.129:36693)
0929 03:15:49.768039 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:49.768042 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:49.768054 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:15:49.768182 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:49.768185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:49.768233 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:49.768312 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:49.768471 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.768966 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.769719 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:49.769932 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:49.769999 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:49.770216 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:49.770636 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:49.770720 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:49.770825 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 03:15:49.770872 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 03:15:49.776705 22086 raft_consensus.cc:1261] T b91e1944461f4b39b1107a4c61542211 P 971840703d2d45d08e9f2727920036c6 [term 1 FOLLOWER]: Refusing update from remote peer a33a5ddf88af461f9ae2e48f79a8ff4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:49.777041 22226 raft_consensus.cc:1261] T b91e1944461f4b39b1107a4c61542211 P 5f130cc04a74491791f8c2b9b5a38f4d [term 1 FOLLOWER]: Refusing update from remote peer a33a5ddf88af461f9ae2e48f79a8ff4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:49.777503 22308 consensus_queue.cc:1035] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:49.777719 22311 consensus_queue.cc:1035] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:49.786844 22315 mvcc.cc:205] Tried to move back new op lower bound from 7334509567078125568 to 7334509567009624064. Current Snapshot: MvccSnapshot[applied={T|T < 7334509567078125568 or (T in {7334509567078125568})}]
I20260929 03:15:49.793023 21787 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:49.793207 21787 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:49.795704 21787 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72: Sending DeleteTablet for 3 replicas of tablet b91e1944461f4b39b1107a4c61542211
I20260929 03:15:49.796442 22206 tablet_service.cc:1514] Processing DeleteTablet for tablet b91e1944461f4b39b1107a4c61542211 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:49 UTC) from {username='oryx', principal='oryx/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:60832
I20260929 03:15:49.796515 21926 tablet_service.cc:1514] Processing DeleteTablet for tablet b91e1944461f4b39b1107a4c61542211 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:49 UTC) from {username='oryx', principal='oryx/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:41114
I20260929 03:15:49.796533 22066 tablet_service.cc:1514] Processing DeleteTablet for tablet b91e1944461f4b39b1107a4c61542211 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:49 UTC) from {username='oryx', principal='oryx/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:33422
I20260929 03:15:49.796950 22324 tablet_replica.cc:330] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a: stopping tablet replica
I20260929 03:15:49.797264 22325 tablet_replica.cc:330] T b91e1944461f4b39b1107a4c61542211 P 971840703d2d45d08e9f2727920036c6: stopping tablet replica
I20260929 03:15:49.797269 22324 raft_consensus.cc:2229] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:15:49.797564 22325 raft_consensus.cc:2229] T b91e1944461f4b39b1107a4c61542211 P 971840703d2d45d08e9f2727920036c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:15:49.797706 22324 raft_consensus.cc:2258] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:49.797972 21787 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 03:15:49.798105 22323 tablet_replica.cc:330] T b91e1944461f4b39b1107a4c61542211 P 5f130cc04a74491791f8c2b9b5a38f4d: stopping tablet replica
W20260929 03:15:49.798406 21787 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:15:49.798416 22323 raft_consensus.cc:2229] T b91e1944461f4b39b1107a4c61542211 P 5f130cc04a74491791f8c2b9b5a38f4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:15:49.798681 22323 raft_consensus.cc:2258] T b91e1944461f4b39b1107a4c61542211 P 5f130cc04a74491791f8c2b9b5a38f4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:49.799170 22325 raft_consensus.cc:2258] T b91e1944461f4b39b1107a4c61542211 P 971840703d2d45d08e9f2727920036c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:49.800009 22324 ts_tablet_manager.cc:1905] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:49.800740 22325 ts_tablet_manager.cc:1905] T b91e1944461f4b39b1107a4c61542211 P 971840703d2d45d08e9f2727920036c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:49.802726 22324 ts_tablet_manager.cc:1918] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:15:49.802809 22324 log.cc:1196] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-0/wal/wals/b91e1944461f4b39b1107a4c61542211
I20260929 03:15:49.803108 22324 ts_tablet_manager.cc:1939] T b91e1944461f4b39b1107a4c61542211 P a33a5ddf88af461f9ae2e48f79a8ff4a: Deleting consensus metadata
I20260929 03:15:49.804178 21926 tablet_service.cc:1467] Processing CreateTablet for tablet ceee20d0c4dc4c18b75e74f13220be29 (DEFAULT_TABLE table=test-table [id=ed4edf63705f426ab9ea4be8a4cf4841]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:49.804385 22206 tablet_service.cc:1467] Processing CreateTablet for tablet ceee20d0c4dc4c18b75e74f13220be29 (DEFAULT_TABLE table=test-table [id=ed4edf63705f426ab9ea4be8a4cf4841]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:49.804425 21926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceee20d0c4dc4c18b75e74f13220be29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:49.804402 22066 tablet_service.cc:1467] Processing CreateTablet for tablet ceee20d0c4dc4c18b75e74f13220be29 (DEFAULT_TABLE table=test-table [id=ed4edf63705f426ab9ea4be8a4cf4841]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:49.804606 22206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceee20d0c4dc4c18b75e74f13220be29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:49.804630 22066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceee20d0c4dc4c18b75e74f13220be29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:49.806667 22300 tablet_bootstrap.cc:492] T ceee20d0c4dc4c18b75e74f13220be29 P 971840703d2d45d08e9f2727920036c6: Bootstrap starting.
I20260929 03:15:49.806771 22302 tablet_bootstrap.cc:492] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a: Bootstrap starting.
I20260929 03:15:49.806959 22301 tablet_bootstrap.cc:492] T ceee20d0c4dc4c18b75e74f13220be29 P 5f130cc04a74491791f8c2b9b5a38f4d: Bootstrap starting.
I20260929 03:15:49.807621 22302 tablet_bootstrap.cc:654] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:49.807621 22300 tablet_bootstrap.cc:654] T ceee20d0c4dc4c18b75e74f13220be29 P 971840703d2d45d08e9f2727920036c6: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:49.807814 22301 tablet_bootstrap.cc:654] T ceee20d0c4dc4c18b75e74f13220be29 P 5f130cc04a74491791f8c2b9b5a38f4d: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:49.808794 22302 tablet_bootstrap.cc:492] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a: No bootstrap required, opened a new log
I20260929 03:15:49.808880 22302 ts_tablet_manager.cc:1397] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:15:49.808863 21775 catalog_manager.cc:4841] TS a33a5ddf88af461f9ae2e48f79a8ff4a (127.13.238.129:36693): tablet b91e1944461f4b39b1107a4c61542211 (table test-table [id=71b47faee93541ccb526f1817601e4d3]) successfully deleted
I20260929 03:15:49.809356 22302 raft_consensus.cc:348] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:49.809464 22302 raft_consensus.cc:374] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:49.809496 22302 raft_consensus.cc:729] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a33a5ddf88af461f9ae2e48f79a8ff4a, State: Initialized, Role: FOLLOWER
I20260929 03:15:49.809612 22302 consensus_queue.cc:260] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:49.809731 22301 tablet_bootstrap.cc:492] T ceee20d0c4dc4c18b75e74f13220be29 P 5f130cc04a74491791f8c2b9b5a38f4d: No bootstrap required, opened a new log
I20260929 03:15:49.809800 22301 ts_tablet_manager.cc:1397] T ceee20d0c4dc4c18b75e74f13220be29 P 5f130cc04a74491791f8c2b9b5a38f4d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260929 03:15:49.809897 22302 ts_tablet_manager.cc:1428] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:49.810266 22301 raft_consensus.cc:348] T ceee20d0c4dc4c18b75e74f13220be29 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:49.810372 22301 raft_consensus.cc:374] T ceee20d0c4dc4c18b75e74f13220be29 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:49.810408 22301 raft_consensus.cc:729] T ceee20d0c4dc4c18b75e74f13220be29 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f130cc04a74491791f8c2b9b5a38f4d, State: Initialized, Role: FOLLOWER
I20260929 03:15:49.810523 22301 consensus_queue.cc:260] T ceee20d0c4dc4c18b75e74f13220be29 P 5f130cc04a74491791f8c2b9b5a38f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:49.810760 22301 ts_tablet_manager.cc:1428] T ceee20d0c4dc4c18b75e74f13220be29 P 5f130cc04a74491791f8c2b9b5a38f4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:49.810858 22323 ts_tablet_manager.cc:1905] T b91e1944461f4b39b1107a4c61542211 P 5f130cc04a74491791f8c2b9b5a38f4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:49.811314 22300 tablet_bootstrap.cc:492] T ceee20d0c4dc4c18b75e74f13220be29 P 971840703d2d45d08e9f2727920036c6: No bootstrap required, opened a new log
I20260929 03:15:49.811400 22300 ts_tablet_manager.cc:1397] T ceee20d0c4dc4c18b75e74f13220be29 P 971840703d2d45d08e9f2727920036c6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 03:15:49.811870 22300 raft_consensus.cc:348] T ceee20d0c4dc4c18b75e74f13220be29 P 971840703d2d45d08e9f2727920036c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:49.811970 22300 raft_consensus.cc:374] T ceee20d0c4dc4c18b75e74f13220be29 P 971840703d2d45d08e9f2727920036c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:49.812008 22300 raft_consensus.cc:729] T ceee20d0c4dc4c18b75e74f13220be29 P 971840703d2d45d08e9f2727920036c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971840703d2d45d08e9f2727920036c6, State: Initialized, Role: FOLLOWER
I20260929 03:15:49.812124 22300 consensus_queue.cc:260] T ceee20d0c4dc4c18b75e74f13220be29 P 971840703d2d45d08e9f2727920036c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:49.812373 22300 ts_tablet_manager.cc:1428] T ceee20d0c4dc4c18b75e74f13220be29 P 971840703d2d45d08e9f2727920036c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:49.813710 22323 ts_tablet_manager.cc:1918] T b91e1944461f4b39b1107a4c61542211 P 5f130cc04a74491791f8c2b9b5a38f4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:15:49.813807 22323 log.cc:1196] T b91e1944461f4b39b1107a4c61542211 P 5f130cc04a74491791f8c2b9b5a38f4d: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/b91e1944461f4b39b1107a4c61542211
I20260929 03:15:49.814090 22323 ts_tablet_manager.cc:1939] T b91e1944461f4b39b1107a4c61542211 P 5f130cc04a74491791f8c2b9b5a38f4d: Deleting consensus metadata
I20260929 03:15:49.814448 22325 ts_tablet_manager.cc:1918] T b91e1944461f4b39b1107a4c61542211 P 971840703d2d45d08e9f2727920036c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:15:49.814543 22325 log.cc:1196] T b91e1944461f4b39b1107a4c61542211 P 971840703d2d45d08e9f2727920036c6: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-1/wal/wals/b91e1944461f4b39b1107a4c61542211
I20260929 03:15:49.814824 22325 ts_tablet_manager.cc:1939] T b91e1944461f4b39b1107a4c61542211 P 971840703d2d45d08e9f2727920036c6: Deleting consensus metadata
I20260929 03:15:49.815418 21773 catalog_manager.cc:4841] TS 5f130cc04a74491791f8c2b9b5a38f4d (127.13.238.131:36921): tablet b91e1944461f4b39b1107a4c61542211 (table test-table [id=71b47faee93541ccb526f1817601e4d3]) successfully deleted
I20260929 03:15:49.815934 21776 catalog_manager.cc:4841] TS 971840703d2d45d08e9f2727920036c6 (127.13.238.130:43511): tablet b91e1944461f4b39b1107a4c61542211 (table test-table [id=71b47faee93541ccb526f1817601e4d3]) successfully deleted
W20260929 03:15:49.898496 22273 tablet.cc:2369] T ceee20d0c4dc4c18b75e74f13220be29 P 5f130cc04a74491791f8c2b9b5a38f4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:15:49.940878 22133 tablet.cc:2369] T ceee20d0c4dc4c18b75e74f13220be29 P 971840703d2d45d08e9f2727920036c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:15:49.972221 22311 raft_consensus.cc:482] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:15:49.972360 22311 raft_consensus.cc:504] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:49.972792 22311 leader_election.cc:288] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 971840703d2d45d08e9f2727920036c6 (127.13.238.130:43511), 5f130cc04a74491791f8c2b9b5a38f4d (127.13.238.131:36921)
I20260929 03:15:49.973227 22086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceee20d0c4dc4c18b75e74f13220be29" candidate_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971840703d2d45d08e9f2727920036c6" is_pre_election: true
I20260929 03:15:49.973240 22226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceee20d0c4dc4c18b75e74f13220be29" candidate_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" is_pre_election: true
I20260929 03:15:49.973381 22086 raft_consensus.cc:2454] T ceee20d0c4dc4c18b75e74f13220be29 P 971840703d2d45d08e9f2727920036c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a33a5ddf88af461f9ae2e48f79a8ff4a in term 0.
I20260929 03:15:49.973381 22226 raft_consensus.cc:2454] T ceee20d0c4dc4c18b75e74f13220be29 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a33a5ddf88af461f9ae2e48f79a8ff4a in term 0.
I20260929 03:15:49.973683 21877 leader_election.cc:302] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f130cc04a74491791f8c2b9b5a38f4d, a33a5ddf88af461f9ae2e48f79a8ff4a; no voters:
I20260929 03:15:49.973858 22311 raft_consensus.cc:2781] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:15:49.973920 22311 raft_consensus.cc:482] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:15:49.973966 22311 raft_consensus.cc:3037] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:49.975034 22311 raft_consensus.cc:504] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:49.975346 22311 leader_election.cc:288] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a [CANDIDATE]: Term 1 election: Requested vote from peers 971840703d2d45d08e9f2727920036c6 (127.13.238.130:43511), 5f130cc04a74491791f8c2b9b5a38f4d (127.13.238.131:36921)
I20260929 03:15:49.975701 22086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceee20d0c4dc4c18b75e74f13220be29" candidate_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971840703d2d45d08e9f2727920036c6"
I20260929 03:15:49.975703 22226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceee20d0c4dc4c18b75e74f13220be29" candidate_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f130cc04a74491791f8c2b9b5a38f4d"
I20260929 03:15:49.975818 22086 raft_consensus.cc:3037] T ceee20d0c4dc4c18b75e74f13220be29 P 971840703d2d45d08e9f2727920036c6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:49.975845 22226 raft_consensus.cc:3037] T ceee20d0c4dc4c18b75e74f13220be29 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:49.976703 22086 raft_consensus.cc:2454] T ceee20d0c4dc4c18b75e74f13220be29 P 971840703d2d45d08e9f2727920036c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a33a5ddf88af461f9ae2e48f79a8ff4a in term 1.
I20260929 03:15:49.976722 22226 raft_consensus.cc:2454] T ceee20d0c4dc4c18b75e74f13220be29 P 5f130cc04a74491791f8c2b9b5a38f4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a33a5ddf88af461f9ae2e48f79a8ff4a in term 1.
I20260929 03:15:49.976976 21880 leader_election.cc:302] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 971840703d2d45d08e9f2727920036c6, a33a5ddf88af461f9ae2e48f79a8ff4a; no voters:
I20260929 03:15:49.977147 22311 raft_consensus.cc:2781] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:49.977233 22311 raft_consensus.cc:686] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 1 LEADER]: Becoming Leader. State: Replica: a33a5ddf88af461f9ae2e48f79a8ff4a, State: Running, Role: LEADER
I20260929 03:15:49.977371 22311 consensus_queue.cc:237] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:49.979072 21789 catalog_manager.cc:5495] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a reported cstate change: term changed from 0 to 1, leader changed from <none> to a33a5ddf88af461f9ae2e48f79a8ff4a (127.13.238.129). New cstate: current_term: 1 leader_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } health_report { overall_health: UNKNOWN } } }
W20260929 03:15:49.984223 21993 tablet.cc:2369] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 03:15:50 dist-test-slave-nbjg krb5kdc[21744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651748, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.190@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM
I20260929 03:15:50.032208 21838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:50.024117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59492 (local address 127.13.238.190:42031)
0929 03:15:50.024288 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:50.024294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:50.024467 (+ 173us) server_negotiation.cc:407] Connection header received
0929 03:15:50.024599 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:50.024602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:50.024648 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:50.024715 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:50.025418 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.025967 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.026800 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.027003 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.027083 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:50.029303 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:50.029307 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:50.029309 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:50.029329 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:50.030481 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:50.030891 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:50.030894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:50.030896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:50.030961 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:50.031228 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:50.031231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:50.031233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:50.031531 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:50.031614 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:50.031909 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 03:15:50.032050 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":63}
I20260929 03:15:50.032217 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:50.024054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42031 (local address 127.0.0.1:59492)
0929 03:15:50.024375 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:50.024387 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:50.024514 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:50.024751 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:50.024754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:50.025208 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:50.025306 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:50.025313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.026069 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.026072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:50.026688 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:50.026698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.027096 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.027099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:50.027225 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:50.027774 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:50.027786 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:50.029179 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:50.030571 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:50.030580 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:50.030587 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:50.030805 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:50.031037 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:50.031040 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:50.031042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:50.031122 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:50.031627 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:50.031634 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:50.031755 (+ 121us) client_negotiation.cc:769] Sending connection context
0929 03:15:50.031852 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 03:15:50.032003 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":35,"thread_start_us":108,"threads_started":1}
I20260929 03:15:50.036008 21789 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:59492:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 03:15:50.036695 21789 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:15:50.041510 21926 tablet_service.cc:1467] Processing CreateTablet for tablet f5f6326731c94b4f8bd4f55382b25904 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=58a7efec28cc4f4a8d5ddbc001c51dec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:15:50.041546 22066 tablet_service.cc:1467] Processing CreateTablet for tablet f5f6326731c94b4f8bd4f55382b25904 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=58a7efec28cc4f4a8d5ddbc001c51dec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:15:50.041697 22206 tablet_service.cc:1467] Processing CreateTablet for tablet f5f6326731c94b4f8bd4f55382b25904 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=58a7efec28cc4f4a8d5ddbc001c51dec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:15:50.041929 21926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5f6326731c94b4f8bd4f55382b25904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:50.041934 22066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5f6326731c94b4f8bd4f55382b25904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:50.042002 22206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5f6326731c94b4f8bd4f55382b25904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:50.044351 22302 tablet_bootstrap.cc:492] T f5f6326731c94b4f8bd4f55382b25904 P a33a5ddf88af461f9ae2e48f79a8ff4a: Bootstrap starting.
I20260929 03:15:50.044697 22300 tablet_bootstrap.cc:492] T f5f6326731c94b4f8bd4f55382b25904 P 971840703d2d45d08e9f2727920036c6: Bootstrap starting.
I20260929 03:15:50.044697 22301 tablet_bootstrap.cc:492] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d: Bootstrap starting.
I20260929 03:15:50.045113 22302 tablet_bootstrap.cc:654] T f5f6326731c94b4f8bd4f55382b25904 P a33a5ddf88af461f9ae2e48f79a8ff4a: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:50.045568 22300 tablet_bootstrap.cc:654] T f5f6326731c94b4f8bd4f55382b25904 P 971840703d2d45d08e9f2727920036c6: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:50.045568 22301 tablet_bootstrap.cc:654] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:50.046818 22300 tablet_bootstrap.cc:492] T f5f6326731c94b4f8bd4f55382b25904 P 971840703d2d45d08e9f2727920036c6: No bootstrap required, opened a new log
I20260929 03:15:50.046903 22300 ts_tablet_manager.cc:1397] T f5f6326731c94b4f8bd4f55382b25904 P 971840703d2d45d08e9f2727920036c6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:15:50.047001 22302 tablet_bootstrap.cc:492] T f5f6326731c94b4f8bd4f55382b25904 P a33a5ddf88af461f9ae2e48f79a8ff4a: No bootstrap required, opened a new log
I20260929 03:15:50.047004 22301 tablet_bootstrap.cc:492] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d: No bootstrap required, opened a new log
I20260929 03:15:50.047075 22301 ts_tablet_manager.cc:1397] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:15:50.047082 22302 ts_tablet_manager.cc:1397] T f5f6326731c94b4f8bd4f55382b25904 P a33a5ddf88af461f9ae2e48f79a8ff4a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 03:15:50.047333 22300 raft_consensus.cc:348] T f5f6326731c94b4f8bd4f55382b25904 P 971840703d2d45d08e9f2727920036c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.047447 22300 raft_consensus.cc:374] T f5f6326731c94b4f8bd4f55382b25904 P 971840703d2d45d08e9f2727920036c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:50.047405 22302 raft_consensus.cc:348] T f5f6326731c94b4f8bd4f55382b25904 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.047492 22300 raft_consensus.cc:729] T f5f6326731c94b4f8bd4f55382b25904 P 971840703d2d45d08e9f2727920036c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971840703d2d45d08e9f2727920036c6, State: Initialized, Role: FOLLOWER
I20260929 03:15:50.047498 22302 raft_consensus.cc:374] T f5f6326731c94b4f8bd4f55382b25904 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:50.047454 22301 raft_consensus.cc:348] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.047528 22302 raft_consensus.cc:729] T f5f6326731c94b4f8bd4f55382b25904 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a33a5ddf88af461f9ae2e48f79a8ff4a, State: Initialized, Role: FOLLOWER
I20260929 03:15:50.047544 22301 raft_consensus.cc:374] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:50.047573 22301 raft_consensus.cc:729] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f130cc04a74491791f8c2b9b5a38f4d, State: Initialized, Role: FOLLOWER
I20260929 03:15:50.047606 22300 consensus_queue.cc:260] T f5f6326731c94b4f8bd4f55382b25904 P 971840703d2d45d08e9f2727920036c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.047650 22302 consensus_queue.cc:260] T f5f6326731c94b4f8bd4f55382b25904 P a33a5ddf88af461f9ae2e48f79a8ff4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.047683 22301 consensus_queue.cc:260] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.047885 22300 ts_tablet_manager.cc:1428] T f5f6326731c94b4f8bd4f55382b25904 P 971840703d2d45d08e9f2727920036c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:50.047904 22302 ts_tablet_manager.cc:1428] T f5f6326731c94b4f8bd4f55382b25904 P a33a5ddf88af461f9ae2e48f79a8ff4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:50.047909 22301 ts_tablet_manager.cc:1428] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:50.047885 22311 tablet_replica.cc:439] T f5f6326731c94b4f8bd4f55382b25904 P a33a5ddf88af461f9ae2e48f79a8ff4a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } } }
I20260929 03:15:50.048012 22311 tablet_replica.cc:442] T f5f6326731c94b4f8bd4f55382b25904 P a33a5ddf88af461f9ae2e48f79a8ff4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:15:50.048090 22307 tablet_replica.cc:439] T f5f6326731c94b4f8bd4f55382b25904 P 971840703d2d45d08e9f2727920036c6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } } }
I20260929 03:15:50.048100 22306 tablet_replica.cc:439] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } } }
I20260929 03:15:50.048225 22307 tablet_replica.cc:442] T f5f6326731c94b4f8bd4f55382b25904 P 971840703d2d45d08e9f2727920036c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:15:50.048225 22306 tablet_replica.cc:442] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:15:50.148654 22306 raft_consensus.cc:482] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:15:50.148801 22306 raft_consensus.cc:504] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.149909 22306 leader_election.cc:288] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a33a5ddf88af461f9ae2e48f79a8ff4a (127.13.238.129:36693), 971840703d2d45d08e9f2727920036c6 (127.13.238.130:43511)
I20260929 03:15:50.154870 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:50.150329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:38453 (local address 127.13.238.130:43511)
0929 03:15:50.150462 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:50.150467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:50.150752 (+ 285us) server_negotiation.cc:407] Connection header received
0929 03:15:50.150924 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:50.150928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:50.150975 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:50.151045 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:50.151276 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.151990 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.153626 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.154257 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.154378 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:50.154430 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:50.154659 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 03:15:50.154732 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:15:50.154870 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:50.150260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:37887 (local address 127.13.238.129:36693)
0929 03:15:50.150404 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:50.150409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:50.150426 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:15:50.150526 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:50.150529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:50.150585 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:50.150660 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:15:50.150962 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.151738 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.153626 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.154264 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.154378 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:50.154430 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:50.154668 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 03:15:50.154732 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:15:50.154902 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:50.150123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:36693 (local address 127.13.238.131:37887)
0929 03:15:50.150297 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:50.150313 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:50.150408 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:50.150711 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:50.150715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:50.150733 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:50.150860 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:50.150866 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.151864 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.151868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:50.153509 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:50.153515 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.154370 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.154374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:50.154517 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:50.154557 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 03:15:50.154634 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 03:15:50.154741 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":104}
I20260929 03:15:50.154914 22339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:50.150123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:43511 (local address 127.13.238.131:38453)
0929 03:15:50.150673 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:50.150685 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:50.150807 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:50.151060 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:50.151062 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:50.151077 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:15:50.151175 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:50.151181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.152082 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.152085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:50.153501 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:50.153513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.154370 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.154374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:50.154514 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:50.154569 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 03:15:50.154634 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 03:15:50.154741 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":315,"thread_start_us":95,"threads_started":1}
I20260929 03:15:50.155464 22086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f6326731c94b4f8bd4f55382b25904" candidate_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971840703d2d45d08e9f2727920036c6" is_pre_election: true
I20260929 03:15:50.155465 21946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f6326731c94b4f8bd4f55382b25904" candidate_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" is_pre_election: true
I20260929 03:15:50.155627 22086 raft_consensus.cc:2454] T f5f6326731c94b4f8bd4f55382b25904 P 971840703d2d45d08e9f2727920036c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f130cc04a74491791f8c2b9b5a38f4d in term 0.
I20260929 03:15:50.155686 21946 raft_consensus.cc:2454] T f5f6326731c94b4f8bd4f55382b25904 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f130cc04a74491791f8c2b9b5a38f4d in term 0.
I20260929 03:15:50.156023 22160 leader_election.cc:302] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f130cc04a74491791f8c2b9b5a38f4d, 971840703d2d45d08e9f2727920036c6; no voters:
I20260929 03:15:50.156292 22306 raft_consensus.cc:2781] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:15:50.156355 22306 raft_consensus.cc:482] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:15:50.156400 22306 raft_consensus.cc:3037] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:50.157361 22306 raft_consensus.cc:504] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.157697 22306 leader_election.cc:288] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d [CANDIDATE]: Term 1 election: Requested vote from peers a33a5ddf88af461f9ae2e48f79a8ff4a (127.13.238.129:36693), 971840703d2d45d08e9f2727920036c6 (127.13.238.130:43511)
I20260929 03:15:50.158037 21946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f6326731c94b4f8bd4f55382b25904" candidate_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a"
I20260929 03:15:50.158041 22086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f6326731c94b4f8bd4f55382b25904" candidate_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971840703d2d45d08e9f2727920036c6"
I20260929 03:15:50.158144 22086 raft_consensus.cc:3037] T f5f6326731c94b4f8bd4f55382b25904 P 971840703d2d45d08e9f2727920036c6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:50.158150 21946 raft_consensus.cc:3037] T f5f6326731c94b4f8bd4f55382b25904 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:50.158993 22086 raft_consensus.cc:2454] T f5f6326731c94b4f8bd4f55382b25904 P 971840703d2d45d08e9f2727920036c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f130cc04a74491791f8c2b9b5a38f4d in term 1.
I20260929 03:15:50.159111 21946 raft_consensus.cc:2454] T f5f6326731c94b4f8bd4f55382b25904 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f130cc04a74491791f8c2b9b5a38f4d in term 1.
I20260929 03:15:50.159258 22160 leader_election.cc:302] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f130cc04a74491791f8c2b9b5a38f4d, 971840703d2d45d08e9f2727920036c6; no voters:
I20260929 03:15:50.159420 22306 raft_consensus.cc:2781] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:50.159627 22306 raft_consensus.cc:686] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d [term 1 LEADER]: Becoming Leader. State: Replica: 5f130cc04a74491791f8c2b9b5a38f4d, State: Running, Role: LEADER
I20260929 03:15:50.159861 22306 consensus_queue.cc:237] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.160974 22340 tablet_replica.cc:439] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d: TxnStatusTablet state changed. Reason: New leader 5f130cc04a74491791f8c2b9b5a38f4d. Latest consensus state: current_term: 1 leader_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } } }
I20260929 03:15:50.161084 22340 tablet_replica.cc:442] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:15:50.161334 22342 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 03:15:50.161487 22342 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 03:15:50.162372 21789 catalog_manager.cc:5495] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f130cc04a74491791f8c2b9b5a38f4d (127.13.238.131). New cstate: current_term: 1 leader_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } health_report { overall_health: HEALTHY } } }
I20260929 03:15:50.176862 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:50.173206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:36921 (local address 127.0.0.1:60844)
0929 03:15:50.173473 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:50.173488 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:50.173602 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:50.173920 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:50.173923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:50.173931 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:15:50.174035 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:50.174041 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.174767 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.174770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:50.175378 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:50.175386 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.175790 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.175794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:50.175881 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:50.175931 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:15:50.176563 (+ 632us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:15:50.176579 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 03:15:50.176643 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 03:15:50.176738 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260929 03:15:50.176882 22298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:50.173299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60844 (local address 127.13.238.131:36921)
0929 03:15:50.173430 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:50.173434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:50.173553 (+ 119us) server_negotiation.cc:407] Connection header received
0929 03:15:50.173740 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:50.173743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:50.173812 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:50.173894 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:50.174133 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.174662 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.175490 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.175699 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.175780 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:50.176063 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:50.176473 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:50.176573 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:50.176654 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 03:15:50.176707 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:15:50.179867 22086 raft_consensus.cc:1261] T f5f6326731c94b4f8bd4f55382b25904 P 971840703d2d45d08e9f2727920036c6 [term 1 FOLLOWER]: Refusing update from remote peer 5f130cc04a74491791f8c2b9b5a38f4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:50.179989 21946 raft_consensus.cc:1261] T f5f6326731c94b4f8bd4f55382b25904 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 1 FOLLOWER]: Refusing update from remote peer 5f130cc04a74491791f8c2b9b5a38f4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:50.180456 22340 consensus_queue.cc:1035] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:50.180637 22306 consensus_queue.cc:1035] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:50.182467 22307 tablet_replica.cc:439] T f5f6326731c94b4f8bd4f55382b25904 P 971840703d2d45d08e9f2727920036c6: TxnStatusTablet state changed. Reason: New leader 5f130cc04a74491791f8c2b9b5a38f4d. Latest consensus state: current_term: 1 leader_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } } }
I20260929 03:15:50.182497 22311 tablet_replica.cc:439] T f5f6326731c94b4f8bd4f55382b25904 P a33a5ddf88af461f9ae2e48f79a8ff4a: TxnStatusTablet state changed. Reason: New leader 5f130cc04a74491791f8c2b9b5a38f4d. Latest consensus state: current_term: 1 leader_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } } }
I20260929 03:15:50.182582 22307 tablet_replica.cc:442] T f5f6326731c94b4f8bd4f55382b25904 P 971840703d2d45d08e9f2727920036c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:15:50.182611 22311 tablet_replica.cc:442] T f5f6326731c94b4f8bd4f55382b25904 P a33a5ddf88af461f9ae2e48f79a8ff4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:15:50.184738 22340 tablet_replica.cc:439] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } } }
I20260929 03:15:50.184863 22340 tablet_replica.cc:442] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:15:50.185745 22307 tablet_replica.cc:439] T f5f6326731c94b4f8bd4f55382b25904 P 971840703d2d45d08e9f2727920036c6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } } }
I20260929 03:15:50.185865 22307 tablet_replica.cc:442] T f5f6326731c94b4f8bd4f55382b25904 P 971840703d2d45d08e9f2727920036c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:15:50.196178 22340 tablet_replica.cc:439] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } } }
I20260929 03:15:50.196312 22340 tablet_replica.cc:442] T f5f6326731c94b4f8bd4f55382b25904 P 5f130cc04a74491791f8c2b9b5a38f4d: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:15:50.197382 22311 tablet_replica.cc:439] T f5f6326731c94b4f8bd4f55382b25904 P a33a5ddf88af461f9ae2e48f79a8ff4a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } } }
I20260929 03:15:50.197495 22311 tablet_replica.cc:442] T f5f6326731c94b4f8bd4f55382b25904 P a33a5ddf88af461f9ae2e48f79a8ff4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:15:50.198122 22314 mvcc.cc:205] Tried to move back new op lower bound from 7334509568731815936 to 7334509568657149952. Current Snapshot: MvccSnapshot[applied={T|T < 7334509568731815936 or (T in {7334509568731815936})}]
I20260929 03:15:50.198807 22298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:50.194896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:39089 (local address 127.13.238.131:36921)
0929 03:15:50.195021 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:50.195026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:50.195252 (+ 226us) server_negotiation.cc:407] Connection header received
0929 03:15:50.195377 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:50.195380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:50.195423 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:50.195481 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:50.195726 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.196510 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.197646 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.197947 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.198017 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:50.198406 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:50.198482 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:50.198540 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:50.198626 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 03:15:50.198675 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:15:50.198851 22349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:50.194805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:36921 (local address 127.13.238.129:39089)
0929 03:15:50.195171 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:50.195182 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:50.195286 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:50.195496 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:50.195498 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:50.195507 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:15:50.195609 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:50.195615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.196637 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.196642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:50.197499 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:50.197510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.198123 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.198126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:50.198232 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:50.198285 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:15:50.198554 (+ 269us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:15:50.198561 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:15:50.198617 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 03:15:50.198699 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":98,"threads_started":1}
I20260929 03:15:50.205914 22226 raft_consensus.cc:1261] T ceee20d0c4dc4c18b75e74f13220be29 P 5f130cc04a74491791f8c2b9b5a38f4d [term 1 FOLLOWER]: Refusing update from remote peer a33a5ddf88af461f9ae2e48f79a8ff4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:50.206277 22086 raft_consensus.cc:1261] T ceee20d0c4dc4c18b75e74f13220be29 P 971840703d2d45d08e9f2727920036c6 [term 1 FOLLOWER]: Refusing update from remote peer a33a5ddf88af461f9ae2e48f79a8ff4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:50.206287 22308 consensus_queue.cc:1035] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:50.206598 22308 consensus_queue.cc:1035] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:50.213013 22316 mvcc.cc:205] Tried to move back new op lower bound from 7334509568840163328 to 7334509567908777984. Current Snapshot: MvccSnapshot[applied={T|T < 7334509568840163328}]
I20260929 03:15:50.229899 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:50.226022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:36693 (local address 127.13.238.131:36183)
0929 03:15:50.226379 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:50.226394 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:50.226480 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:50.226700 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:50.226703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:50.226710 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:15:50.226851 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:50.226859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.227861 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.227865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:50.228585 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:50.228596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.229145 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.229147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:50.229218 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:50.229269 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:15:50.229574 (+ 305us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:15:50.229582 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 03:15:50.229658 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 03:15:50.229737 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":102,"threads_started":1}
I20260929 03:15:50.229964 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:50.226324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:36183 (local address 127.13.238.129:36693)
0929 03:15:50.226448 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:50.226452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:50.226468 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:15:50.226552 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:50.226555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:50.226604 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:50.226672 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:50.226961 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.227767 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.228696 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.229047 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.229147 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:50.229384 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:50.229469 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:50.229544 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:50.229748 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 03:15:50.229813 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:15:50.255236 21789 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:50.255385 21789 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:50.257301 21789 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72: Sending DeleteTablet for 3 replicas of tablet ceee20d0c4dc4c18b75e74f13220be29
I20260929 03:15:50.257762 21926 tablet_service.cc:1514] Processing DeleteTablet for tablet ceee20d0c4dc4c18b75e74f13220be29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:50 UTC) from {username='oryx', principal='oryx/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:41114
I20260929 03:15:50.257953 22324 tablet_replica.cc:330] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a: stopping tablet replica
I20260929 03:15:50.257943 22066 tablet_service.cc:1514] Processing DeleteTablet for tablet ceee20d0c4dc4c18b75e74f13220be29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:50 UTC) from {username='oryx', principal='oryx/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:33422
I20260929 03:15:50.258082 22325 tablet_replica.cc:330] T ceee20d0c4dc4c18b75e74f13220be29 P 971840703d2d45d08e9f2727920036c6: stopping tablet replica
I20260929 03:15:50.258076 22324 raft_consensus.cc:2229] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:15:50.258172 22325 raft_consensus.cc:2229] T ceee20d0c4dc4c18b75e74f13220be29 P 971840703d2d45d08e9f2727920036c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:15:50.258170 22206 tablet_service.cc:1514] Processing DeleteTablet for tablet ceee20d0c4dc4c18b75e74f13220be29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:50 UTC) from {username='oryx', principal='oryx/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:60832
I20260929 03:15:50.258296 22325 raft_consensus.cc:2258] T ceee20d0c4dc4c18b75e74f13220be29 P 971840703d2d45d08e9f2727920036c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:50.258307 22324 raft_consensus.cc:2258] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:50.258359 22323 tablet_replica.cc:330] T ceee20d0c4dc4c18b75e74f13220be29 P 5f130cc04a74491791f8c2b9b5a38f4d: stopping tablet replica
I20260929 03:15:50.258457 22323 raft_consensus.cc:2229] T ceee20d0c4dc4c18b75e74f13220be29 P 5f130cc04a74491791f8c2b9b5a38f4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:15:50.258553 22323 raft_consensus.cc:2258] T ceee20d0c4dc4c18b75e74f13220be29 P 5f130cc04a74491791f8c2b9b5a38f4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:50.259184 22323 ts_tablet_manager.cc:1905] T ceee20d0c4dc4c18b75e74f13220be29 P 5f130cc04a74491791f8c2b9b5a38f4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:50.259224 22324 ts_tablet_manager.cc:1905] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:50.259284 22325 ts_tablet_manager.cc:1905] T ceee20d0c4dc4c18b75e74f13220be29 P 971840703d2d45d08e9f2727920036c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:50.261371 22323 ts_tablet_manager.cc:1918] T ceee20d0c4dc4c18b75e74f13220be29 P 5f130cc04a74491791f8c2b9b5a38f4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:15:50.261436 22323 log.cc:1196] T ceee20d0c4dc4c18b75e74f13220be29 P 5f130cc04a74491791f8c2b9b5a38f4d: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/ceee20d0c4dc4c18b75e74f13220be29
I20260929 03:15:50.261637 22323 ts_tablet_manager.cc:1939] T ceee20d0c4dc4c18b75e74f13220be29 P 5f130cc04a74491791f8c2b9b5a38f4d: Deleting consensus metadata
I20260929 03:15:50.261788 22324 ts_tablet_manager.cc:1918] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:15:50.261788 22325 ts_tablet_manager.cc:1918] T ceee20d0c4dc4c18b75e74f13220be29 P 971840703d2d45d08e9f2727920036c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:15:50.261864 22325 log.cc:1196] T ceee20d0c4dc4c18b75e74f13220be29 P 971840703d2d45d08e9f2727920036c6: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-1/wal/wals/ceee20d0c4dc4c18b75e74f13220be29
I20260929 03:15:50.261864 22324 log.cc:1196] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-0/wal/wals/ceee20d0c4dc4c18b75e74f13220be29
I20260929 03:15:50.262099 22324 ts_tablet_manager.cc:1939] T ceee20d0c4dc4c18b75e74f13220be29 P a33a5ddf88af461f9ae2e48f79a8ff4a: Deleting consensus metadata
I20260929 03:15:50.262099 22325 ts_tablet_manager.cc:1939] T ceee20d0c4dc4c18b75e74f13220be29 P 971840703d2d45d08e9f2727920036c6: Deleting consensus metadata
I20260929 03:15:50.275822 21776 catalog_manager.cc:4841] TS 971840703d2d45d08e9f2727920036c6 (127.13.238.130:43511): tablet ceee20d0c4dc4c18b75e74f13220be29 (table test-table [id=ed4edf63705f426ab9ea4be8a4cf4841]) successfully deleted
I20260929 03:15:50.276221 21773 catalog_manager.cc:4841] TS 5f130cc04a74491791f8c2b9b5a38f4d (127.13.238.131:36921): tablet ceee20d0c4dc4c18b75e74f13220be29 (table test-table [id=ed4edf63705f426ab9ea4be8a4cf4841]) successfully deleted
I20260929 03:15:50.276224 21775 catalog_manager.cc:4841] TS a33a5ddf88af461f9ae2e48f79a8ff4a (127.13.238.129:36693): tablet ceee20d0c4dc4c18b75e74f13220be29 (table test-table [id=ed4edf63705f426ab9ea4be8a4cf4841]) successfully deleted
Sep 29 03:15:50 dist-test-slave-nbjg krb5kdc[21744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:15:50 dist-test-slave-nbjg krb5kdc[21744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:15:50 dist-test-slave-nbjg krb5kdc[21744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:15:50.292532 21838 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:50.286063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59506 (local address 127.13.238.190:42031)
0929 03:15:50.286237 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:50.286244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:50.286392 (+ 148us) server_negotiation.cc:407] Connection header received
0929 03:15:50.286519 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:50.286522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:50.286574 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:50.286664 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:50.287318 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.288112 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.288950 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.289315 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.289419 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:50.292406 (+ 2987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59506: BlockingRecv error: recv got EOF from 127.0.0.1:59506 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 03:15:50 dist-test-slave-nbjg krb5kdc[21744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM
I20260929 03:15:50.302620 21838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:50.293963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59518 (local address 127.13.238.190:42031)
0929 03:15:50.294080 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:50.294085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:50.294332 (+ 247us) server_negotiation.cc:407] Connection header received
0929 03:15:50.294459 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:50.294461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:50.294503 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:50.294561 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:50.295272 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.295766 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.296880 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.297080 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.297146 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:50.299708 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:50.299711 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:50.299713 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:50.299737 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:50.300977 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:50.301466 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:50.301469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:50.301470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:50.301568 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:50.301818 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:50.301820 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:50.301821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:50.301988 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:50.302105 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:50.302342 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 03:15:50.302475 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 03:15:50.305043 21789 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:15:50.305461 21789 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:15:50.310017 21926 tablet_service.cc:1467] Processing CreateTablet for tablet ca668c9b551c42898c06477a7a3101a0 (DEFAULT_TABLE table=test-table [id=ad3cbc0c505441c9b86b30c497e15ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:50.310370 21926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca668c9b551c42898c06477a7a3101a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:50.310309 22066 tablet_service.cc:1467] Processing CreateTablet for tablet ca668c9b551c42898c06477a7a3101a0 (DEFAULT_TABLE table=test-table [id=ad3cbc0c505441c9b86b30c497e15ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:50.310567 22066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca668c9b551c42898c06477a7a3101a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:50.312851 22302 tablet_bootstrap.cc:492] T ca668c9b551c42898c06477a7a3101a0 P a33a5ddf88af461f9ae2e48f79a8ff4a: Bootstrap starting.
I20260929 03:15:50.312887 22300 tablet_bootstrap.cc:492] T ca668c9b551c42898c06477a7a3101a0 P 971840703d2d45d08e9f2727920036c6: Bootstrap starting.
I20260929 03:15:50.313668 22300 tablet_bootstrap.cc:654] T ca668c9b551c42898c06477a7a3101a0 P 971840703d2d45d08e9f2727920036c6: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:50.313778 22302 tablet_bootstrap.cc:654] T ca668c9b551c42898c06477a7a3101a0 P a33a5ddf88af461f9ae2e48f79a8ff4a: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:50.314910 22300 tablet_bootstrap.cc:492] T ca668c9b551c42898c06477a7a3101a0 P 971840703d2d45d08e9f2727920036c6: No bootstrap required, opened a new log
I20260929 03:15:50.314917 22206 tablet_service.cc:1467] Processing CreateTablet for tablet ca668c9b551c42898c06477a7a3101a0 (DEFAULT_TABLE table=test-table [id=ad3cbc0c505441c9b86b30c497e15ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:50.315006 22300 ts_tablet_manager.cc:1397] T ca668c9b551c42898c06477a7a3101a0 P 971840703d2d45d08e9f2727920036c6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:15:50.315158 22206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca668c9b551c42898c06477a7a3101a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:50.315503 22300 raft_consensus.cc:348] T ca668c9b551c42898c06477a7a3101a0 P 971840703d2d45d08e9f2727920036c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.315624 22300 raft_consensus.cc:374] T ca668c9b551c42898c06477a7a3101a0 P 971840703d2d45d08e9f2727920036c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:50.315659 22300 raft_consensus.cc:729] T ca668c9b551c42898c06477a7a3101a0 P 971840703d2d45d08e9f2727920036c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971840703d2d45d08e9f2727920036c6, State: Initialized, Role: FOLLOWER
I20260929 03:15:50.315797 22300 consensus_queue.cc:260] T ca668c9b551c42898c06477a7a3101a0 P 971840703d2d45d08e9f2727920036c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.316087 22300 ts_tablet_manager.cc:1428] T ca668c9b551c42898c06477a7a3101a0 P 971840703d2d45d08e9f2727920036c6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 03:15:50.316885 22302 tablet_bootstrap.cc:492] T ca668c9b551c42898c06477a7a3101a0 P a33a5ddf88af461f9ae2e48f79a8ff4a: No bootstrap required, opened a new log
I20260929 03:15:50.316983 22302 ts_tablet_manager.cc:1397] T ca668c9b551c42898c06477a7a3101a0 P a33a5ddf88af461f9ae2e48f79a8ff4a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 03:15:50.317416 22302 raft_consensus.cc:348] T ca668c9b551c42898c06477a7a3101a0 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.317524 22302 raft_consensus.cc:374] T ca668c9b551c42898c06477a7a3101a0 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:50.317555 22302 raft_consensus.cc:729] T ca668c9b551c42898c06477a7a3101a0 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a33a5ddf88af461f9ae2e48f79a8ff4a, State: Initialized, Role: FOLLOWER
I20260929 03:15:50.317696 22302 consensus_queue.cc:260] T ca668c9b551c42898c06477a7a3101a0 P a33a5ddf88af461f9ae2e48f79a8ff4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.317965 22302 ts_tablet_manager.cc:1428] T ca668c9b551c42898c06477a7a3101a0 P a33a5ddf88af461f9ae2e48f79a8ff4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:50.318190 22301 tablet_bootstrap.cc:492] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d: Bootstrap starting.
I20260929 03:15:50.319238 22301 tablet_bootstrap.cc:654] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:50.320536 22301 tablet_bootstrap.cc:492] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d: No bootstrap required, opened a new log
I20260929 03:15:50.320627 22301 ts_tablet_manager.cc:1397] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:15:50.321096 22301 raft_consensus.cc:348] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.321254 22301 raft_consensus.cc:374] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:50.321300 22301 raft_consensus.cc:729] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f130cc04a74491791f8c2b9b5a38f4d, State: Initialized, Role: FOLLOWER
I20260929 03:15:50.321437 22301 consensus_queue.cc:260] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.321964 22301 ts_tablet_manager.cc:1428] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:50.734571 22350 raft_consensus.cc:482] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:15:50.734755 22350 raft_consensus.cc:504] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.735195 22350 leader_election.cc:288] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a33a5ddf88af461f9ae2e48f79a8ff4a (127.13.238.129:36693), 971840703d2d45d08e9f2727920036c6 (127.13.238.130:43511)
I20260929 03:15:50.735646 22086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca668c9b551c42898c06477a7a3101a0" candidate_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971840703d2d45d08e9f2727920036c6" is_pre_election: true
I20260929 03:15:50.735653 21946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca668c9b551c42898c06477a7a3101a0" candidate_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" is_pre_election: true
I20260929 03:15:50.735785 21946 raft_consensus.cc:2454] T ca668c9b551c42898c06477a7a3101a0 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f130cc04a74491791f8c2b9b5a38f4d in term 0.
I20260929 03:15:50.735816 22086 raft_consensus.cc:2454] T ca668c9b551c42898c06477a7a3101a0 P 971840703d2d45d08e9f2727920036c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f130cc04a74491791f8c2b9b5a38f4d in term 0.
I20260929 03:15:50.736083 22159 leader_election.cc:302] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f130cc04a74491791f8c2b9b5a38f4d, a33a5ddf88af461f9ae2e48f79a8ff4a; no voters:
I20260929 03:15:50.736274 22350 raft_consensus.cc:2781] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:15:50.736332 22350 raft_consensus.cc:482] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:15:50.736369 22350 raft_consensus.cc:3037] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:50.737368 22350 raft_consensus.cc:504] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.737735 22350 leader_election.cc:288] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d [CANDIDATE]: Term 1 election: Requested vote from peers a33a5ddf88af461f9ae2e48f79a8ff4a (127.13.238.129:36693), 971840703d2d45d08e9f2727920036c6 (127.13.238.130:43511)
I20260929 03:15:50.738099 21946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca668c9b551c42898c06477a7a3101a0" candidate_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a"
I20260929 03:15:50.738106 22086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca668c9b551c42898c06477a7a3101a0" candidate_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971840703d2d45d08e9f2727920036c6"
I20260929 03:15:50.738238 21946 raft_consensus.cc:3037] T ca668c9b551c42898c06477a7a3101a0 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:50.738252 22086 raft_consensus.cc:3037] T ca668c9b551c42898c06477a7a3101a0 P 971840703d2d45d08e9f2727920036c6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:50.739253 21946 raft_consensus.cc:2454] T ca668c9b551c42898c06477a7a3101a0 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f130cc04a74491791f8c2b9b5a38f4d in term 1.
I20260929 03:15:50.739259 22086 raft_consensus.cc:2454] T ca668c9b551c42898c06477a7a3101a0 P 971840703d2d45d08e9f2727920036c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f130cc04a74491791f8c2b9b5a38f4d in term 1.
I20260929 03:15:50.739545 22159 leader_election.cc:302] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f130cc04a74491791f8c2b9b5a38f4d, a33a5ddf88af461f9ae2e48f79a8ff4a; no voters:
I20260929 03:15:50.739758 22306 raft_consensus.cc:2781] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:50.739859 22306 raft_consensus.cc:686] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d [term 1 LEADER]: Becoming Leader. State: Replica: 5f130cc04a74491791f8c2b9b5a38f4d, State: Running, Role: LEADER
I20260929 03:15:50.739993 22306 consensus_queue.cc:237] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.741843 21787 catalog_manager.cc:5495] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f130cc04a74491791f8c2b9b5a38f4d (127.13.238.131). New cstate: current_term: 1 leader_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } health_report { overall_health: HEALTHY } } }
I20260929 03:15:50.778924 22374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:50.775798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60860 (local address 127.13.238.131:36921)
0929 03:15:50.776064 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:50.776068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:50.776080 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:15:50.776125 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:50.776127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:50.776171 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:50.776262 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:50.776515 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.777028 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.777887 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.778110 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.778248 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:50.778483 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:50.778558 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:50.778618 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:50.778714 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 03:15:50.778772 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":125,"threads_started":1}
I20260929 03:15:50.781333 22086 raft_consensus.cc:1261] T ca668c9b551c42898c06477a7a3101a0 P 971840703d2d45d08e9f2727920036c6 [term 1 FOLLOWER]: Refusing update from remote peer 5f130cc04a74491791f8c2b9b5a38f4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:50.781333 21946 raft_consensus.cc:1261] T ca668c9b551c42898c06477a7a3101a0 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 1 FOLLOWER]: Refusing update from remote peer 5f130cc04a74491791f8c2b9b5a38f4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:50.781826 22306 consensus_queue.cc:1035] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:50.782013 22340 consensus_queue.cc:1035] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:50.792460 21789 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:50.792601 21789 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:50.794620 21789 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72: Sending DeleteTablet for 3 replicas of tablet ca668c9b551c42898c06477a7a3101a0
I20260929 03:15:50.795197 22066 tablet_service.cc:1514] Processing DeleteTablet for tablet ca668c9b551c42898c06477a7a3101a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:50 UTC) from {username='oryx', principal='oryx/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:33422
I20260929 03:15:50.795205 21926 tablet_service.cc:1514] Processing DeleteTablet for tablet ca668c9b551c42898c06477a7a3101a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:50 UTC) from {username='oryx', principal='oryx/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:41114
I20260929 03:15:50.795526 22384 tablet_replica.cc:330] T ca668c9b551c42898c06477a7a3101a0 P 971840703d2d45d08e9f2727920036c6: stopping tablet replica
I20260929 03:15:50.795694 22384 raft_consensus.cc:2229] T ca668c9b551c42898c06477a7a3101a0 P 971840703d2d45d08e9f2727920036c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:15:50.795740 22206 tablet_service.cc:1514] Processing DeleteTablet for tablet ca668c9b551c42898c06477a7a3101a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:50 UTC) from {username='oryx', principal='oryx/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:60832
I20260929 03:15:50.795948 22384 raft_consensus.cc:2258] T ca668c9b551c42898c06477a7a3101a0 P 971840703d2d45d08e9f2727920036c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:50.796180 22385 tablet_replica.cc:330] T ca668c9b551c42898c06477a7a3101a0 P a33a5ddf88af461f9ae2e48f79a8ff4a: stopping tablet replica
I20260929 03:15:50.796311 22385 raft_consensus.cc:2229] T ca668c9b551c42898c06477a7a3101a0 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:15:50.796446 22385 raft_consensus.cc:2258] T ca668c9b551c42898c06477a7a3101a0 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:50.796419 21789 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 03:15:50.796657 22386 tablet_replica.cc:330] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d: stopping tablet replica
I20260929 03:15:50.796783 22386 raft_consensus.cc:2229] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d [term 1 LEADER]: Raft consensus shutting down.
W20260929 03:15:50.796821 21789 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:15:50.797088 22384 ts_tablet_manager.cc:1905] T ca668c9b551c42898c06477a7a3101a0 P 971840703d2d45d08e9f2727920036c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:50.797104 22386 raft_consensus.cc:2258] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:50.798496 22386 ts_tablet_manager.cc:1905] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:50.799947 22384 ts_tablet_manager.cc:1918] T ca668c9b551c42898c06477a7a3101a0 P 971840703d2d45d08e9f2727920036c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:15:50.800086 22384 log.cc:1196] T ca668c9b551c42898c06477a7a3101a0 P 971840703d2d45d08e9f2727920036c6: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-1/wal/wals/ca668c9b551c42898c06477a7a3101a0
I20260929 03:15:50.800395 22384 ts_tablet_manager.cc:1939] T ca668c9b551c42898c06477a7a3101a0 P 971840703d2d45d08e9f2727920036c6: Deleting consensus metadata
I20260929 03:15:50.801038 22386 ts_tablet_manager.cc:1918] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:15:50.801111 22386 log.cc:1196] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/ca668c9b551c42898c06477a7a3101a0
I20260929 03:15:50.801407 22386 ts_tablet_manager.cc:1939] T ca668c9b551c42898c06477a7a3101a0 P 5f130cc04a74491791f8c2b9b5a38f4d: Deleting consensus metadata
I20260929 03:15:50.801537 21776 catalog_manager.cc:4841] TS 971840703d2d45d08e9f2727920036c6 (127.13.238.130:43511): tablet ca668c9b551c42898c06477a7a3101a0 (table test-table [id=ad3cbc0c505441c9b86b30c497e15ad4]) successfully deleted
I20260929 03:15:50.801636 21926 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa77b26cb1149af8488f88460fb48e0 (DEFAULT_TABLE table=test-table [id=03b273f8e423419aac70dbc0fa3c26bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:50.801836 22066 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa77b26cb1149af8488f88460fb48e0 (DEFAULT_TABLE table=test-table [id=03b273f8e423419aac70dbc0fa3c26bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:50.801889 21926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fa77b26cb1149af8488f88460fb48e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:50.802057 22066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fa77b26cb1149af8488f88460fb48e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:50.803864 22300 tablet_bootstrap.cc:492] T 9fa77b26cb1149af8488f88460fb48e0 P 971840703d2d45d08e9f2727920036c6: Bootstrap starting.
I20260929 03:15:50.804193 22302 tablet_bootstrap.cc:492] T 9fa77b26cb1149af8488f88460fb48e0 P a33a5ddf88af461f9ae2e48f79a8ff4a: Bootstrap starting.
I20260929 03:15:50.804174 21773 catalog_manager.cc:4841] TS 5f130cc04a74491791f8c2b9b5a38f4d (127.13.238.131:36921): tablet ca668c9b551c42898c06477a7a3101a0 (table test-table [id=ad3cbc0c505441c9b86b30c497e15ad4]) successfully deleted
I20260929 03:15:50.804661 22300 tablet_bootstrap.cc:654] T 9fa77b26cb1149af8488f88460fb48e0 P 971840703d2d45d08e9f2727920036c6: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:50.804970 22302 tablet_bootstrap.cc:654] T 9fa77b26cb1149af8488f88460fb48e0 P a33a5ddf88af461f9ae2e48f79a8ff4a: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:50.805456 22206 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa77b26cb1149af8488f88460fb48e0 (DEFAULT_TABLE table=test-table [id=03b273f8e423419aac70dbc0fa3c26bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:15:50.805689 22206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fa77b26cb1149af8488f88460fb48e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:50.805972 22385 ts_tablet_manager.cc:1905] T ca668c9b551c42898c06477a7a3101a0 P a33a5ddf88af461f9ae2e48f79a8ff4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:50.807857 22301 tablet_bootstrap.cc:492] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d: Bootstrap starting.
I20260929 03:15:50.808169 22385 ts_tablet_manager.cc:1918] T ca668c9b551c42898c06477a7a3101a0 P a33a5ddf88af461f9ae2e48f79a8ff4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:15:50.808251 22385 log.cc:1196] T ca668c9b551c42898c06477a7a3101a0 P a33a5ddf88af461f9ae2e48f79a8ff4a: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-0/wal/wals/ca668c9b551c42898c06477a7a3101a0
I20260929 03:15:50.808480 22385 ts_tablet_manager.cc:1939] T ca668c9b551c42898c06477a7a3101a0 P a33a5ddf88af461f9ae2e48f79a8ff4a: Deleting consensus metadata
I20260929 03:15:50.808635 22301 tablet_bootstrap.cc:654] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:50.809150 21775 catalog_manager.cc:4841] TS a33a5ddf88af461f9ae2e48f79a8ff4a (127.13.238.129:36693): tablet ca668c9b551c42898c06477a7a3101a0 (table test-table [id=ad3cbc0c505441c9b86b30c497e15ad4]) successfully deleted
I20260929 03:15:50.809906 22302 tablet_bootstrap.cc:492] T 9fa77b26cb1149af8488f88460fb48e0 P a33a5ddf88af461f9ae2e48f79a8ff4a: No bootstrap required, opened a new log
I20260929 03:15:50.809937 22301 tablet_bootstrap.cc:492] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d: No bootstrap required, opened a new log
I20260929 03:15:50.810005 22301 ts_tablet_manager.cc:1397] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:15:50.810005 22302 ts_tablet_manager.cc:1397] T 9fa77b26cb1149af8488f88460fb48e0 P a33a5ddf88af461f9ae2e48f79a8ff4a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260929 03:15:50.810410 22300 tablet_bootstrap.cc:492] T 9fa77b26cb1149af8488f88460fb48e0 P 971840703d2d45d08e9f2727920036c6: No bootstrap required, opened a new log
I20260929 03:15:50.810420 22302 raft_consensus.cc:348] T 9fa77b26cb1149af8488f88460fb48e0 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.810485 22300 ts_tablet_manager.cc:1397] T 9fa77b26cb1149af8488f88460fb48e0 P 971840703d2d45d08e9f2727920036c6: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260929 03:15:50.810424 22301 raft_consensus.cc:348] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.810516 22302 raft_consensus.cc:374] T 9fa77b26cb1149af8488f88460fb48e0 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:50.810523 22301 raft_consensus.cc:374] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:50.810545 22302 raft_consensus.cc:729] T 9fa77b26cb1149af8488f88460fb48e0 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a33a5ddf88af461f9ae2e48f79a8ff4a, State: Initialized, Role: FOLLOWER
I20260929 03:15:50.810551 22301 raft_consensus.cc:729] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f130cc04a74491791f8c2b9b5a38f4d, State: Initialized, Role: FOLLOWER
I20260929 03:15:50.810657 22301 consensus_queue.cc:260] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.810657 22302 consensus_queue.cc:260] T 9fa77b26cb1149af8488f88460fb48e0 P a33a5ddf88af461f9ae2e48f79a8ff4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.810902 22301 ts_tablet_manager.cc:1428] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:50.810899 22302 ts_tablet_manager.cc:1428] T 9fa77b26cb1149af8488f88460fb48e0 P a33a5ddf88af461f9ae2e48f79a8ff4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:50.810928 22300 raft_consensus.cc:348] T 9fa77b26cb1149af8488f88460fb48e0 P 971840703d2d45d08e9f2727920036c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.811023 22300 raft_consensus.cc:374] T 9fa77b26cb1149af8488f88460fb48e0 P 971840703d2d45d08e9f2727920036c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:50.811050 22300 raft_consensus.cc:729] T 9fa77b26cb1149af8488f88460fb48e0 P 971840703d2d45d08e9f2727920036c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971840703d2d45d08e9f2727920036c6, State: Initialized, Role: FOLLOWER
I20260929 03:15:50.811157 22300 consensus_queue.cc:260] T 9fa77b26cb1149af8488f88460fb48e0 P 971840703d2d45d08e9f2727920036c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.811393 22300 ts_tablet_manager.cc:1428] T 9fa77b26cb1149af8488f88460fb48e0 P 971840703d2d45d08e9f2727920036c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:15:50.948858 22340 raft_consensus.cc:482] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:15:50.948994 22340 raft_consensus.cc:504] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.949441 22340 leader_election.cc:288] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a33a5ddf88af461f9ae2e48f79a8ff4a (127.13.238.129:36693), 971840703d2d45d08e9f2727920036c6 (127.13.238.130:43511)
I20260929 03:15:50.949955 21946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa77b26cb1149af8488f88460fb48e0" candidate_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" is_pre_election: true
I20260929 03:15:50.949955 22086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa77b26cb1149af8488f88460fb48e0" candidate_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971840703d2d45d08e9f2727920036c6" is_pre_election: true
I20260929 03:15:50.950116 22086 raft_consensus.cc:2454] T 9fa77b26cb1149af8488f88460fb48e0 P 971840703d2d45d08e9f2727920036c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f130cc04a74491791f8c2b9b5a38f4d in term 0.
I20260929 03:15:50.950116 21946 raft_consensus.cc:2454] T 9fa77b26cb1149af8488f88460fb48e0 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f130cc04a74491791f8c2b9b5a38f4d in term 0.
I20260929 03:15:50.950449 22159 leader_election.cc:302] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f130cc04a74491791f8c2b9b5a38f4d, a33a5ddf88af461f9ae2e48f79a8ff4a; no voters:
I20260929 03:15:50.950615 22340 raft_consensus.cc:2781] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:15:50.950680 22340 raft_consensus.cc:482] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:15:50.950706 22340 raft_consensus.cc:3037] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:50.951737 22340 raft_consensus.cc:504] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.952056 22340 leader_election.cc:288] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d [CANDIDATE]: Term 1 election: Requested vote from peers a33a5ddf88af461f9ae2e48f79a8ff4a (127.13.238.129:36693), 971840703d2d45d08e9f2727920036c6 (127.13.238.130:43511)
I20260929 03:15:50.952422 22086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa77b26cb1149af8488f88460fb48e0" candidate_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971840703d2d45d08e9f2727920036c6"
I20260929 03:15:50.952441 21946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa77b26cb1149af8488f88460fb48e0" candidate_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a"
I20260929 03:15:50.952548 22086 raft_consensus.cc:3037] T 9fa77b26cb1149af8488f88460fb48e0 P 971840703d2d45d08e9f2727920036c6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:50.952556 21946 raft_consensus.cc:3037] T 9fa77b26cb1149af8488f88460fb48e0 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:50.953372 22086 raft_consensus.cc:2454] T 9fa77b26cb1149af8488f88460fb48e0 P 971840703d2d45d08e9f2727920036c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f130cc04a74491791f8c2b9b5a38f4d in term 1.
I20260929 03:15:50.953472 21946 raft_consensus.cc:2454] T 9fa77b26cb1149af8488f88460fb48e0 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f130cc04a74491791f8c2b9b5a38f4d in term 1.
I20260929 03:15:50.953630 22160 leader_election.cc:302] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f130cc04a74491791f8c2b9b5a38f4d, 971840703d2d45d08e9f2727920036c6; no voters:
I20260929 03:15:50.953815 22340 raft_consensus.cc:2781] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:50.953887 22340 raft_consensus.cc:686] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d [term 1 LEADER]: Becoming Leader. State: Replica: 5f130cc04a74491791f8c2b9b5a38f4d, State: Running, Role: LEADER
I20260929 03:15:50.954023 22340 consensus_queue.cc:237] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } }
I20260929 03:15:50.955611 21786 catalog_manager.cc:5495] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f130cc04a74491791f8c2b9b5a38f4d (127.13.238.131). New cstate: current_term: 1 leader_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f130cc04a74491791f8c2b9b5a38f4d" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36921 } health_report { overall_health: HEALTHY } } }
I20260929 03:15:50.977952 22374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:50.974948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:41067 (local address 127.13.238.131:36921)
0929 03:15:50.975064 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:50.975069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:50.975216 (+ 147us) server_negotiation.cc:407] Connection header received
0929 03:15:50.975330 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:50.975332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:50.975373 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:50.975431 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:15:50.975689 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.976188 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.977004 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.977220 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.977293 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:50.977499 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:15:50.977565 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:15:50.977625 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:50.977768 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 03:15:50.977827 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":38}
I20260929 03:15:50.977967 22393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:50.974896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:36921 (local address 127.13.238.131:41067)
0929 03:15:50.975138 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:50.975153 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:50.975253 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:50.975452 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:50.975455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:50.975462 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:15:50.975563 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:50.975568 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.976283 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.976286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:50.976902 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:50.976911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:50.977324 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:50.977327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:50.977388 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:50.977415 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:15:50.977696 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:15:50.977702 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:15:50.977755 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 03:15:50.977830 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":97,"threads_started":1}
I20260929 03:15:50.983726 22086 raft_consensus.cc:1261] T 9fa77b26cb1149af8488f88460fb48e0 P 971840703d2d45d08e9f2727920036c6 [term 1 FOLLOWER]: Refusing update from remote peer 5f130cc04a74491791f8c2b9b5a38f4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:50.983749 21946 raft_consensus.cc:1261] T 9fa77b26cb1149af8488f88460fb48e0 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 1 FOLLOWER]: Refusing update from remote peer 5f130cc04a74491791f8c2b9b5a38f4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:15:50.984071 22306 consensus_queue.cc:1035] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "971840703d2d45d08e9f2727920036c6" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:50.984238 22340 consensus_queue.cc:1035] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a33a5ddf88af461f9ae2e48f79a8ff4a" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 36693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:15:51.016130 21787 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:51.016321 21787 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:15:51.018290 21787 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f3c46901f2847f38c9ff7db5f86fe72: Sending DeleteTablet for 3 replicas of tablet 9fa77b26cb1149af8488f88460fb48e0
I20260929 03:15:51.018867 21926 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fa77b26cb1149af8488f88460fb48e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:51 UTC) from {username='oryx', principal='oryx/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:41114
I20260929 03:15:51.018877 22066 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fa77b26cb1149af8488f88460fb48e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:51 UTC) from {username='oryx', principal='oryx/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:33422
I20260929 03:15:51.018898 22206 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fa77b26cb1149af8488f88460fb48e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:15:51 UTC) from {username='oryx', principal='oryx/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:60832
I20260929 03:15:51.019047 22386 tablet_replica.cc:330] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d: stopping tablet replica
I20260929 03:15:51.019057 22384 tablet_replica.cc:330] T 9fa77b26cb1149af8488f88460fb48e0 P 971840703d2d45d08e9f2727920036c6: stopping tablet replica
I20260929 03:15:51.019152 22386 raft_consensus.cc:2229] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:15:51.019182 22385 tablet_replica.cc:330] T 9fa77b26cb1149af8488f88460fb48e0 P a33a5ddf88af461f9ae2e48f79a8ff4a: stopping tablet replica
I20260929 03:15:51.019179 22384 raft_consensus.cc:2229] T 9fa77b26cb1149af8488f88460fb48e0 P 971840703d2d45d08e9f2727920036c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:15:51.019287 22384 raft_consensus.cc:2258] T 9fa77b26cb1149af8488f88460fb48e0 P 971840703d2d45d08e9f2727920036c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:51.019291 22385 raft_consensus.cc:2229] T 9fa77b26cb1149af8488f88460fb48e0 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:15:51.019351 22386 raft_consensus.cc:2258] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:51.019389 22385 raft_consensus.cc:2258] T 9fa77b26cb1149af8488f88460fb48e0 P a33a5ddf88af461f9ae2e48f79a8ff4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:15:51.019620 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 21861
I20260929 03:15:51.020058 22386 ts_tablet_manager.cc:1905] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:51.021301 22384 ts_tablet_manager.cc:1905] T 9fa77b26cb1149af8488f88460fb48e0 P 971840703d2d45d08e9f2727920036c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:15:51.022809 22386 ts_tablet_manager.cc:1918] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:15:51.022893 22386 log.cc:1196] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/9fa77b26cb1149af8488f88460fb48e0
I20260929 03:15:51.023144 22386 ts_tablet_manager.cc:1939] T 9fa77b26cb1149af8488f88460fb48e0 P 5f130cc04a74491791f8c2b9b5a38f4d: Deleting consensus metadata
I20260929 03:15:51.023681 22384 ts_tablet_manager.cc:1918] T 9fa77b26cb1149af8488f88460fb48e0 P 971840703d2d45d08e9f2727920036c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:15:51.023773 22384 log.cc:1196] T 9fa77b26cb1149af8488f88460fb48e0 P 971840703d2d45d08e9f2727920036c6: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790651713532531-14266-0/minicluster-data/ts-1/wal/wals/9fa77b26cb1149af8488f88460fb48e0
I20260929 03:15:51.023885 21773 catalog_manager.cc:4841] TS 5f130cc04a74491791f8c2b9b5a38f4d (127.13.238.131:36921): tablet 9fa77b26cb1149af8488f88460fb48e0 (table test-table [id=03b273f8e423419aac70dbc0fa3c26bf]) successfully deleted
I20260929 03:15:51.024006 22384 ts_tablet_manager.cc:1939] T 9fa77b26cb1149af8488f88460fb48e0 P 971840703d2d45d08e9f2727920036c6: Deleting consensus metadata
I20260929 03:15:51.024554 21776 catalog_manager.cc:4841] TS 971840703d2d45d08e9f2727920036c6 (127.13.238.130:43511): tablet 9fa77b26cb1149af8488f88460fb48e0 (table test-table [id=03b273f8e423419aac70dbc0fa3c26bf]) successfully deleted
W20260929 03:15:51.027871 21775 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.238.129:36693 (error 108)
I20260929 03:15:51.028059 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 22001
I20260929 03:15:51.029291 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:51.028921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:36693 (local address 127.0.0.1:41122)
0929 03:15:51.029126 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:51.029198 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.129:36693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":52,"threads_started":1}
W20260929 03:15:51.029430 21775 catalog_manager.cc:4584] TS a33a5ddf88af461f9ae2e48f79a8ff4a (127.13.238.129:36693): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9fa77b26cb1149af8488f88460fb48e0: Network error: Client connection negotiation failed: client connection to 127.13.238.129:36693: connect: Connection refused (error 111)
I20260929 03:15:51.033748 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 22141
I20260929 03:15:51.040009 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 21757
2026-09-29T03:15:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3327 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:15:51 dist-test-slave-nbjg krb5kdc[22401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:15:51 dist-test-slave-nbjg krb5kdc[22401](info): set up 2 sockets
Sep 29 03:15:51 dist-test-slave-nbjg krb5kdc[22401](info): commencing operation
krb5kdc: starting...
W20260929 03:15:52.002868 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:15:52 dist-test-slave-nbjg krb5kdc[22401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:15:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:15:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:15:52.120592 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:41581
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:41943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:41581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:52.230219 22414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:52.230512 22414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:52.230572 22414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:52.234552 22414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:52.234608 22414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:52.234633 22414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:52.234663 22414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:52.234691 22414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:15:52.239945 22414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:41943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:41581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:41581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:52.241000 22414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:52.242015 22414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:52.250471 22420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:52.250527 22422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:52.250471 22419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:52.251068 22414 server_base.cc:1034] running on GCE node
I20260929 03:15:52.251753 22414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:52.252969 22414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:52.254148 22414 hybrid_clock.cc:648] HybridClock initialized: now 1790651752254127 us; error 34 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T03:15:52Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1218 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:15:52 dist-test-slave-nbjg krb5kdc[22427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:15:52 dist-test-slave-nbjg krb5kdc[22427](info): set up 2 sockets
Sep 29 03:15:52 dist-test-slave-nbjg krb5kdc[22427](info): commencing operation
krb5kdc: starting...
W20260929 03:15:53.218811 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:15:53 dist-test-slave-nbjg krb5kdc[22427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:15:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:15:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:15:53.336651 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:44891
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:34121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:44891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:53.446154 22440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:53.446449 22440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:53.446493 22440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:53.450491 22440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:53.450556 22440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:53.450575 22440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:53.450590 22440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:53.450603 22440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:15:53.455988 22440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:44891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:44891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:53.457178 22440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:53.458278 22440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:53.467013 22446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:53.467025 22448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:53.467025 22445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:53.467381 22440 server_base.cc:1034] running on GCE node
I20260929 03:15:53.467960 22440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:53.469143 22440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:53.470286 22440 hybrid_clock.cc:648] HybridClock initialized: now 1790651753470265 us; error 36 us; skew 500 ppm
Sep 29 03:15:53 dist-test-slave-nbjg krb5kdc[22427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651753, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:53.473073 22440 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:15:53.474120 22440 webserver.cc:460] Webserver started at http://127.13.238.190:38351/ using document root <none> and password file <none>
I20260929 03:15:53.474727 22440 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:53.474788 22440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:53.475000 22440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:53.477180 22440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "66ee5cf02e774095a7305a270b66769a"
format_stamp: "Formatted at 2026-09-29 03:15:53 on dist-test-slave-nbjg"
server_key: "ffa93ef0ee203d50b086f424b118c6cd"
server_key_iv: "4eebb84b691944ea3446e88f6baaf21a"
server_key_version: "encryptionkey@0"
I20260929 03:15:53.477816 22440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "66ee5cf02e774095a7305a270b66769a"
format_stamp: "Formatted at 2026-09-29 03:15:53 on dist-test-slave-nbjg"
server_key: "ffa93ef0ee203d50b086f424b118c6cd"
server_key_iv: "4eebb84b691944ea3446e88f6baaf21a"
server_key_version: "encryptionkey@0"
I20260929 03:15:53.481758 22440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:15:53.484366 22455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:53.485735 22440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:15:53.485877 22440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "66ee5cf02e774095a7305a270b66769a"
format_stamp: "Formatted at 2026-09-29 03:15:53 on dist-test-slave-nbjg"
server_key: "ffa93ef0ee203d50b086f424b118c6cd"
server_key_iv: "4eebb84b691944ea3446e88f6baaf21a"
server_key_version: "encryptionkey@0"
I20260929 03:15:53.485980 22440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:53.510380 22440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:53.513667 22440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:53.513866 22440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:53.520751 22440 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:44891
I20260929 03:15:53.520763 22507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:44891 every 8 connection(s)
I20260929 03:15:53.521763 22440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:15:53.522979 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 22440
I20260929 03:15:53.523101 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:15:53.523418 14266 external_mini_cluster.cc:1428] Setting key d58314dac40a177a9aacde0e9b32ece7
I20260929 03:15:53.525224 22508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:15:53.531900 22508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a: Bootstrap starting.
Sep 29 03:15:53 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:53.534592 22508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a: Neither blocks nor log segments found. Creating new log.
I20260929 03:15:53.535398 22508 log.cc:824] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:53.537925 22508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a: No bootstrap required, opened a new log
I20260929 03:15:53.538563 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:53.524942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36172 (local address 127.13.238.190:44891)
0929 03:15:53.525388 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:53.525404 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:53.525443 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:15:53.526085 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:53.526111 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:53.526504 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:53.526796 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:53.527661 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:53.528703 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:53.529861 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:53.530463 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:53.530599 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:53.533468 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:53.533474 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:53.533491 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:53.533524 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:53.535286 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:53.535808 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:53.535816 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:53.535823 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:53.535988 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:53.536349 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:53.536352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:53.536353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:53.536789 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:53.536942 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:53.537318 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 03:15:53.537699 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":138,"threads_started":1}
I20260929 03:15:53.541885 22508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ee5cf02e774095a7305a270b66769a" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 44891 } }
I20260929 03:15:53.542133 22508 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:15:53.542219 22508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ee5cf02e774095a7305a270b66769a, State: Initialized, Role: FOLLOWER
I20260929 03:15:53.542649 22508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ee5cf02e774095a7305a270b66769a" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 44891 } }
I20260929 03:15:53.542773 22508 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:15:53.542831 22508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:15:53.542924 22508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:15:53.543857 22508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ee5cf02e774095a7305a270b66769a" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 44891 } }
I20260929 03:15:53.544198 22508 leader_election.cc:302] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66ee5cf02e774095a7305a270b66769a; no voters:
I20260929 03:15:53.544492 22508 leader_election.cc:288] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:15:53.544615 22513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:15:53.544826 22513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [term 1 LEADER]: Becoming Leader. State: Replica: 66ee5cf02e774095a7305a270b66769a, State: Running, Role: LEADER
I20260929 03:15:53.545125 22513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ee5cf02e774095a7305a270b66769a" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 44891 } }
I20260929 03:15:53.545693 22508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:53.546816 22515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66ee5cf02e774095a7305a270b66769a. Latest consensus state: current_term: 1 leader_uuid: "66ee5cf02e774095a7305a270b66769a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ee5cf02e774095a7305a270b66769a" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 44891 } } }
I20260929 03:15:53.546954 22515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:53.547137 22514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66ee5cf02e774095a7305a270b66769a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ee5cf02e774095a7305a270b66769a" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 44891 } } }
I20260929 03:15:53.547282 22514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:53.547364 22522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:15:53.550709 22522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:15:53.556226 22522 catalog_manager.cc:1347] Generated new cluster ID: a1ca728f6bc648aba7430c91d6909ec9
I20260929 03:15:53.556300 22522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:15:53.576794 22522 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:15:53.577962 22522 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:15:53.587387 22522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a: Generated new TSK 0
I20260929 03:15:53.588135 22522 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
I20260929 03:15:53.627705 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:0
--local_ip_for_outbound_sockets=127.13.238.129
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:44891
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:34121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:53.740166 22534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:53.740454 22534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:53.740509 22534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:53.744639 22534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:53.744709 22534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:53.744793 22534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:15:53.750658 22534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:53.751780 22534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:53.752799 22534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:53.762116 22542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:53.762171 22539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:53.762367 22534 server_base.cc:1034] running on GCE node
W20260929 03:15:53.762117 22540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:53.762866 22534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:53.763473 22534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:53.764678 22534 hybrid_clock.cc:648] HybridClock initialized: now 1790651753764623 us; error 62 us; skew 500 ppm
Sep 29 03:15:53 dist-test-slave-nbjg krb5kdc[22427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651753, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:53.767418 22534 init.cc:377] Logged in from keytab as kudu/127.13.238.129@KRBTEST.COM (short username kudu)
I20260929 03:15:53.768275 22534 webserver.cc:460] Webserver started at http://127.13.238.129:38733/ using document root <none> and password file <none>
I20260929 03:15:53.768862 22534 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:53.768929 22534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:53.769109 22534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:53.771169 22534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/data/instance:
uuid: "9cb161cb90aa4289968f59097c5abd57"
format_stamp: "Formatted at 2026-09-29 03:15:53 on dist-test-slave-nbjg"
server_key: "a61db6bb38677672c08139944f173790"
server_key_iv: "9749c800e0ce42db4c5f7c5aef8cb325"
server_key_version: "encryptionkey@0"
I20260929 03:15:53.771711 22534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance:
uuid: "9cb161cb90aa4289968f59097c5abd57"
format_stamp: "Formatted at 2026-09-29 03:15:53 on dist-test-slave-nbjg"
server_key: "a61db6bb38677672c08139944f173790"
server_key_iv: "9749c800e0ce42db4c5f7c5aef8cb325"
server_key_version: "encryptionkey@0"
I20260929 03:15:53.775442 22534 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:15:53.777904 22549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:53.779175 22534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:53.779304 22534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "9cb161cb90aa4289968f59097c5abd57"
format_stamp: "Formatted at 2026-09-29 03:15:53 on dist-test-slave-nbjg"
server_key: "a61db6bb38677672c08139944f173790"
server_key_iv: "9749c800e0ce42db4c5f7c5aef8cb325"
server_key_version: "encryptionkey@0"
I20260929 03:15:53.779399 22534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:53.792470 22534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:53.795487 22534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:53.795668 22534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:53.796223 22534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:53.797039 22534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:53.797084 22534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:53.797135 22534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:53.797163 22534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:53.805395 22534 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:41279
I20260929 03:15:53.805440 22662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:41279 every 8 connection(s)
I20260929 03:15:53.806358 22534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:15:53.814477 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 22534
I20260929 03:15:53.814606 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance
I20260929 03:15:53.814932 14266 external_mini_cluster.cc:1428] Setting key 8c379c91124d5c58eaab13be653d1dba
Sep 29 03:15:53 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651753, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:53.820371 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:53.808214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:34761 (local address 127.13.238.190:44891)
0929 03:15:53.808385 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:53.808389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:53.809451 (+ 1062us) server_negotiation.cc:407] Connection header received
0929 03:15:53.810380 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:53.810384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:53.810432 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:53.810495 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:53.811703 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:53.812237 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:53.813494 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:53.813742 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:53.813823 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:53.816607 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:53.816612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:53.816615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:53.816642 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:53.818033 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:53.818641 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:53.818644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:53.818645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:53.818750 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:53.819211 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:53.819214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:53.819216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:53.819411 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:53.819492 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:53.820067 (+ 575us) server_negotiation.cc:299] Negotiation successful
0929 03:15:53.820208 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 03:15:53.821204 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:53.808548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.129:34761)
0929 03:15:53.809311 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:53.809355 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:53.810127 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:53.810651 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:53.810657 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:53.811014 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:53.811516 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:53.811535 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:53.812432 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:53.812435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:53.813370 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:53.813380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:53.813872 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:53.813874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:53.814016 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:53.814703 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:53.814725 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:53.816436 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:53.818147 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:53.818155 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:53.818174 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:53.818544 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:53.818842 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:53.818845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:53.818847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:53.819038 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:53.819503 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:53.819509 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:53.819788 (+ 279us) client_negotiation.cc:769] Sending connection context
0929 03:15:53.819952 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 03:15:53.820239 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":126,"threads_started":1}
I20260929 03:15:53.822592 22663 heartbeater.cc:347] Connected to a master server at 127.13.238.190:44891
I20260929 03:15:53.822870 22663 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:53.823391 22663 heartbeater.cc:511] Master 127.13.238.190:44891 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.238.130@KRBTEST.COM; defaulting to no policy
I20260929 03:15:53.825067 22472 ts_manager.cc:194] Registered new tserver with Master: 9cb161cb90aa4289968f59097c5abd57 (127.13.238.129:41279)
I20260929 03:15:53.826342 22472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.129@KRBTEST.COM'} at 127.13.238.129:34761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
I20260929 03:15:53.841508 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:0
--local_ip_for_outbound_sockets=127.13.238.130
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:44891
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:34121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:53.951272 22668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:53.951545 22668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:53.951589 22668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:53.955649 22668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:53.955714 22668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:53.955775 22668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:15:53.961333 22668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:53.962448 22668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:53.963476 22668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:53.973018 22674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:53.973084 22668 server_base.cc:1034] running on GCE node
W20260929 03:15:53.973126 22673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:53.973034 22676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:53.973583 22668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:53.974246 22668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:53.975399 22668 hybrid_clock.cc:648] HybridClock initialized: now 1790651753975375 us; error 33 us; skew 500 ppm
Sep 29 03:15:53 dist-test-slave-nbjg krb5kdc[22427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651753, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:53.978071 22668 init.cc:377] Logged in from keytab as kudu/127.13.238.130@KRBTEST.COM (short username kudu)
I20260929 03:15:53.979000 22668 webserver.cc:460] Webserver started at http://127.13.238.130:37341/ using document root <none> and password file <none>
I20260929 03:15:53.979560 22668 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:53.979622 22668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:53.979863 22668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:53.981959 22668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/data/instance:
uuid: "4081660706624c59931cd0c4f66068ad"
format_stamp: "Formatted at 2026-09-29 03:15:53 on dist-test-slave-nbjg"
server_key: "bb171152f4e75406258758d7f7d6f97e"
server_key_iv: "9856f63c91b19d0df0651f3b588e8a59"
server_key_version: "encryptionkey@0"
I20260929 03:15:53.982511 22668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance:
uuid: "4081660706624c59931cd0c4f66068ad"
format_stamp: "Formatted at 2026-09-29 03:15:53 on dist-test-slave-nbjg"
server_key: "bb171152f4e75406258758d7f7d6f97e"
server_key_iv: "9856f63c91b19d0df0651f3b588e8a59"
server_key_version: "encryptionkey@0"
I20260929 03:15:53.986268 22668 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:15:53.988667 22683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:53.990000 22668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:15:53.990120 22668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "4081660706624c59931cd0c4f66068ad"
format_stamp: "Formatted at 2026-09-29 03:15:53 on dist-test-slave-nbjg"
server_key: "bb171152f4e75406258758d7f7d6f97e"
server_key_iv: "9856f63c91b19d0df0651f3b588e8a59"
server_key_version: "encryptionkey@0"
I20260929 03:15:53.990254 22668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:54.004040 22668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:54.007221 22668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:54.007400 22668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:54.008029 22668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:54.008889 22668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:54.008937 22668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:54.008978 22668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:54.009006 22668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:54.017526 22668 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:41417
I20260929 03:15:54.017560 22796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:41417 every 8 connection(s)
I20260929 03:15:54.018471 22668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:15:54.027817 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 22668
I20260929 03:15:54.027941 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance
Sep 29 03:15:54 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651753, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:54.028213 14266 external_mini_cluster.cc:1428] Setting key 913d3b78decd7e2c0fad72fdddfcd354
I20260929 03:15:54.032123 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:54.020444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:51153 (local address 127.13.238.190:44891)
0929 03:15:54.020648 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:54.020652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:54.021750 (+ 1098us) server_negotiation.cc:407] Connection header received
0929 03:15:54.022681 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:54.022684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:54.022740 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:54.022835 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:54.024081 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.024633 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.025946 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.026169 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.026274 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:54.028836 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:54.028839 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:54.028841 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:54.028866 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:54.030177 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:54.030690 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:54.030694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:54.030696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:54.030773 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:54.031077 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:54.031080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:54.031082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:54.031262 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:54.031343 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:54.031845 (+ 502us) server_negotiation.cc:299] Negotiation successful
0929 03:15:54.031964 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 03:15:54.033039 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:54.020878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.130:51153)
0929 03:15:54.021578 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:54.021656 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:54.022464 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:54.023000 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:54.023007 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:54.023377 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:54.023925 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:54.023939 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.024786 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.024796 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:54.025816 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:54.025827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.026342 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.026346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:54.026492 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:54.027079 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:54.027095 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:54.028624 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:54.030307 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:54.030313 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:54.030331 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:54.030595 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:54.030852 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:54.030855 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:54.030857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:54.030994 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:54.031357 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:54.031364 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:54.031610 (+ 246us) client_negotiation.cc:769] Sending connection context
0929 03:15:54.031788 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 03:15:54.032114 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":125,"threads_started":1}
I20260929 03:15:54.034185 22797 heartbeater.cc:347] Connected to a master server at 127.13.238.190:44891
I20260929 03:15:54.034438 22797 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:54.034907 22797 heartbeater.cc:511] Master 127.13.238.190:44891 requested a full tablet report, sending...
I20260929 03:15:54.035925 22472 ts_manager.cc:194] Registered new tserver with Master: 4081660706624c59931cd0c4f66068ad (127.13.238.130:41417)
I20260929 03:15:54.036700 22472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.130@KRBTEST.COM'} at 127.13.238.130:51153
WARNING: no policy specified for kudu/127.13.238.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
I20260929 03:15:54.053445 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:0
--local_ip_for_outbound_sockets=127.13.238.131
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:44891
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:34121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:54.162945 22802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:54.163246 22802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:54.163290 22802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:54.167380 22802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:54.167445 22802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:54.167510 22802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:15:54.172948 22802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:54.174029 22802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:54.175092 22802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:54.184544 22810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:54.184563 22808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:54.184514 22807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:54.184724 22802 server_base.cc:1034] running on GCE node
I20260929 03:15:54.185245 22802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:54.185864 22802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:54.187027 22802 hybrid_clock.cc:648] HybridClock initialized: now 1790651754187012 us; error 24 us; skew 500 ppm
Sep 29 03:15:54 dist-test-slave-nbjg krb5kdc[22427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651754, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:54.189721 22802 init.cc:377] Logged in from keytab as kudu/127.13.238.131@KRBTEST.COM (short username kudu)
I20260929 03:15:54.190696 22802 webserver.cc:460] Webserver started at http://127.13.238.131:37575/ using document root <none> and password file <none>
I20260929 03:15:54.191234 22802 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:54.191290 22802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:54.191453 22802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:15:54.193485 22802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/data/instance:
uuid: "61b3440ce1e94ba59ba219aa9b8ec676"
format_stamp: "Formatted at 2026-09-29 03:15:54 on dist-test-slave-nbjg"
server_key: "c043d79fc04eef060b59317d17f0ed89"
server_key_iv: "97791778c4bf412687de54efc3653e0b"
server_key_version: "encryptionkey@0"
I20260929 03:15:54.193994 22802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance:
uuid: "61b3440ce1e94ba59ba219aa9b8ec676"
format_stamp: "Formatted at 2026-09-29 03:15:54 on dist-test-slave-nbjg"
server_key: "c043d79fc04eef060b59317d17f0ed89"
server_key_iv: "97791778c4bf412687de54efc3653e0b"
server_key_version: "encryptionkey@0"
I20260929 03:15:54.198087 22802 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260929 03:15:54.200613 22817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:54.201901 22802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:15:54.201997 22802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "61b3440ce1e94ba59ba219aa9b8ec676"
format_stamp: "Formatted at 2026-09-29 03:15:54 on dist-test-slave-nbjg"
server_key: "c043d79fc04eef060b59317d17f0ed89"
server_key_iv: "97791778c4bf412687de54efc3653e0b"
server_key_version: "encryptionkey@0"
I20260929 03:15:54.202081 22802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:54.217113 22802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:54.220265 22802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:54.220418 22802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:54.221006 22802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:54.221791 22802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:54.221836 22802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:54.221875 22802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:54.221894 22802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:54.230361 22802 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:42753
I20260929 03:15:54.230440 22930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:42753 every 8 connection(s)
I20260929 03:15:54.231276 22802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:15:54.239743 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 22802
I20260929 03:15:54.239856 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance
I20260929 03:15:54.240142 14266 external_mini_cluster.cc:1428] Setting key ea69fdb5ea64c52c21731b573ddac7a3
Sep 29 03:15:54 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651754, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:15:54.244251 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:54.233103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:57789 (local address 127.13.238.190:44891)
0929 03:15:54.233248 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:54.233252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:54.234412 (+ 1160us) server_negotiation.cc:407] Connection header received
0929 03:15:54.235299 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:54.235302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:54.235358 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:54.235432 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:54.236608 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.237137 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.238319 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.238540 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.238628 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:54.241020 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:54.241024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:54.241027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:54.241056 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:54.242553 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:54.243001 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:54.243003 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:54.243005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:54.243075 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:54.243328 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:54.243330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:54.243331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:54.243489 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:54.243556 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:54.244033 (+ 477us) server_negotiation.cc:299] Negotiation successful
0929 03:15:54.244135 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:15:54.245016 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:54.233508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.131:57789)
0929 03:15:54.234225 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:54.234275 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:54.235089 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:54.235613 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:54.235621 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:54.235980 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:54.236456 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:54.236469 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.237275 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.237279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:54.238168 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:54.238177 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.238679 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.238683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:54.238857 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:54.239400 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:54.239416 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:54.240875 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:15:54.242652 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:54.242657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:54.242670 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:54.242922 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:54.243141 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:15:54.243143 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:15:54.243144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:15:54.243266 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:15:54.243558 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:15:54.243563 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:15:54.243789 (+ 226us) client_negotiation.cc:769] Sending connection context
0929 03:15:54.243965 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 03:15:54.244221 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":107,"threads_started":1}
I20260929 03:15:54.246119 22931 heartbeater.cc:347] Connected to a master server at 127.13.238.190:44891
I20260929 03:15:54.246399 22931 heartbeater.cc:464] Registering TS with master...
I20260929 03:15:54.246846 22931 heartbeater.cc:511] Master 127.13.238.190:44891 requested a full tablet report, sending...
I20260929 03:15:54.247828 22472 ts_manager.cc:194] Registered new tserver with Master: 61b3440ce1e94ba59ba219aa9b8ec676 (127.13.238.131:42753)
I20260929 03:15:54.248361 22472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.131@KRBTEST.COM'} at 127.13.238.131:57789
I20260929 03:15:54.254173 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.13.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.129.keytab.
Entry for principal oryx/127.13.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.129.keytab.
WARNING: no policy specified for oryx/127.13.238.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.238.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.130.keytab.
Entry for principal oryx/127.13.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.130.keytab.
WARNING: no policy specified for oryx/127.13.238.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.131.keytab.
Entry for principal oryx/127.13.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.131.keytab.
I20260929 03:15:54.327183 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 22534
I20260929 03:15:54.332139 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 22668
I20260929 03:15:54.336889 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 22802
I20260929 03:15:54.341444 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 22440
I20260929 03:15:54.347151 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:44891
--webserver_interface=127.13.238.190
--webserver_port=38351
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:34121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:44891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:15:54.456173 22940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:54.456444 22940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:54.456498 22940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:54.460574 22940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:15:54.460644 22940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:54.460668 22940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:54.460690 22940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:15:54.460711 22940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:15:54.466138 22940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:44891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:44891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=38351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:54.467301 22940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:54.468377 22940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:54.477016 22948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:54.477000 22940 server_base.cc:1034] running on GCE node
W20260929 03:15:54.476979 22945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:54.476972 22946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:54.477792 22940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:54.478929 22940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:54.480106 22940 hybrid_clock.cc:648] HybridClock initialized: now 1790651754480081 us; error 41 us; skew 500 ppm
Sep 29 03:15:54 dist-test-slave-nbjg krb5kdc[22427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651754, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:54.482884 22940 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:15:54.483779 22940 webserver.cc:460] Webserver started at http://127.13.238.190:38351/ using document root <none> and password file <none>
I20260929 03:15:54.484323 22940 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:54.484404 22940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:54.488291 22940 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:15:54.490408 22955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:54.491569 22940 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:15:54.491681 22940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "66ee5cf02e774095a7305a270b66769a"
format_stamp: "Formatted at 2026-09-29 03:15:53 on dist-test-slave-nbjg"
server_key: "ffa93ef0ee203d50b086f424b118c6cd"
server_key_iv: "4eebb84b691944ea3446e88f6baaf21a"
server_key_version: "encryptionkey@0"
I20260929 03:15:54.492237 22940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:54.504182 22940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:54.507119 22940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:54.507296 22940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:54.514147 22940 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:44891
I20260929 03:15:54.514160 23007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:44891 every 8 connection(s)
I20260929 03:15:54.515231 22940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:15:54.519634 23008 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:15:54.521353 23008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a: Bootstrap starting.
I20260929 03:15:54.523027 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 22940
I20260929 03:15:54.523435 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:41279
--local_ip_for_outbound_sockets=127.13.238.129
--tserver_master_addrs=127.13.238.190:44891
--webserver_port=38733
--webserver_interface=127.13.238.129
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:34121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.238.129
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:15:54.537721 23008 log.cc:824] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a: Log is configured to *not* fsync() on all Append() calls
I20260929 03:15:54.544600 23008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:15:54.544936 23008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a: Bootstrap complete.
I20260929 03:15:54.548966 23008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ee5cf02e774095a7305a270b66769a" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 44891 } }
I20260929 03:15:54.549363 23008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ee5cf02e774095a7305a270b66769a, State: Initialized, Role: FOLLOWER
I20260929 03:15:54.549917 23008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ee5cf02e774095a7305a270b66769a" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 44891 } }
I20260929 03:15:54.550050 23008 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:15:54.550122 23008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:15:54.550238 23008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:15:54.552251 23008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ee5cf02e774095a7305a270b66769a" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 44891 } }
I20260929 03:15:54.552603 23008 leader_election.cc:302] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66ee5cf02e774095a7305a270b66769a; no voters:
I20260929 03:15:54.552932 23008 leader_election.cc:288] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 03:15:54.553074 23012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:15:54.553290 23012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [term 2 LEADER]: Becoming Leader. State: Replica: 66ee5cf02e774095a7305a270b66769a, State: Running, Role: LEADER
I20260929 03:15:54.553668 23012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ee5cf02e774095a7305a270b66769a" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 44891 } }
I20260929 03:15:54.554275 23008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:15:54.555064 23014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66ee5cf02e774095a7305a270b66769a. Latest consensus state: current_term: 2 leader_uuid: "66ee5cf02e774095a7305a270b66769a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ee5cf02e774095a7305a270b66769a" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 44891 } } }
I20260929 03:15:54.555192 23014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:54.555244 23013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "66ee5cf02e774095a7305a270b66769a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ee5cf02e774095a7305a270b66769a" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 44891 } } }
I20260929 03:15:54.555341 23013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a [sys.catalog]: This master's current role is: LEADER
I20260929 03:15:54.555692 23021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:15:54.559224 23021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:15:54.559942 23021 catalog_manager.cc:1259] Loaded cluster ID: a1ca728f6bc648aba7430c91d6909ec9
I20260929 03:15:54.560001 23021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:15:54.562330 23021 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:15:54.563019 23021 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 66ee5cf02e774095a7305a270b66769a: Loaded TSK: 0
I20260929 03:15:54.564286 23021 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:15:54.639256 23010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:54.639523 23010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:54.639567 23010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:54.643777 23010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:54.643836 23010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:54.643895 23010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:15:54.649679 23010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.129.keytab
--principal=oryx/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:41279
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=38733
--tserver_master_addrs=127.13.238.190:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:54.651253 23010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:54.652678 23010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:54.665210 23035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:54.665200 23034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:54.665797 23037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:54.666222 23010 server_base.cc:1034] running on GCE node
I20260929 03:15:54.666698 23010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:54.667508 23010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:54.668668 23010 hybrid_clock.cc:648] HybridClock initialized: now 1790651754668623 us; error 59 us; skew 500 ppm
Sep 29 03:15:54 dist-test-slave-nbjg krb5kdc[22427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651754, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:54.672231 23010 init.cc:377] Logged in from keytab as oryx/127.13.238.129@KRBTEST.COM (short username oryx)
I20260929 03:15:54.673964 23010 webserver.cc:460] Webserver started at http://127.13.238.129:38733/ using document root <none> and password file <none>
I20260929 03:15:54.674724 23010 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:54.674851 23010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:54.680505 23010 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:15:54.683274 23044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:54.685084 23010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:15:54.685319 23010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "9cb161cb90aa4289968f59097c5abd57"
format_stamp: "Formatted at 2026-09-29 03:15:53 on dist-test-slave-nbjg"
server_key: "a61db6bb38677672c08139944f173790"
server_key_iv: "9749c800e0ce42db4c5f7c5aef8cb325"
server_key_version: "encryptionkey@0"
I20260929 03:15:54.686175 23010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:54.721915 23010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:54.724903 23010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:54.725075 23010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:54.725690 23010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:54.726567 23010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:54.726615 23010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:54.726657 23010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:54.726684 23010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:54.735353 23010 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:41279
I20260929 03:15:54.735388 23157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:41279 every 8 connection(s)
I20260929 03:15:54.736373 23010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:15:54.740971 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 23010
I20260929 03:15:54.741456 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:41417
--local_ip_for_outbound_sockets=127.13.238.130
--tserver_master_addrs=127.13.238.190:44891
--webserver_port=37341
--webserver_interface=127.13.238.130
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:34121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.238.130
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:15:54 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:15:54 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:15:54.749663 23160 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:54.738568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:60361 (local address 127.13.238.190:44891)
0929 03:15:54.738988 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:54.739004 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:54.739514 (+ 510us) server_negotiation.cc:407] Connection header received
0929 03:15:54.740549 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:54.740561 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:54.740958 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:54.741345 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:54.742612 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.743597 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.744797 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.745274 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.745427 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:54.748963 (+ 3536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.129:60361: BlockingRecv error: recv got EOF from 127.13.238.129:60361 (error 108)
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":91,"threads_started":1}
I20260929 03:15:54.749758 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:54.738635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.129:60361)
0929 03:15:54.739347 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:54.739400 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:54.740194 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:54.741459 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:54.741466 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:54.741926 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:54.742424 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:54.742438 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.743740 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.743743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:54.744673 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:54.744686 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.745434 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.745439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:54.745614 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:54.746229 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:54.746259 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:54.748756 (+ 2497us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":117,"threads_started":1}
W20260929 03:15:54.750309 23158 heartbeater.cc:644] Failed to heartbeat to 127.13.238.190:44891 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.238.190:44891: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Sep 29 03:15:54 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:15:54 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:15:54.755635 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:54.750782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.129:50599)
0929 03:15:54.750892 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:54.750906 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:54.750986 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:54.751204 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:54.751207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:54.751355 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:54.751444 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:54.751449 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.752142 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.752145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:54.752683 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:54.752691 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.753101 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.753104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:54.753186 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:54.753606 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:54.753617 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:54.755504 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 03:15:54.755874 23160 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:54.750869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:50599 (local address 127.13.238.190:44891)
0929 03:15:54.750970 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:54.750974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:54.750986 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:15:54.751070 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:54.751074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:54.751117 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:54.751181 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:54.751546 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.752047 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.752790 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.753006 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.753090 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:54.755718 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.129:50599: BlockingRecv error: recv got EOF from 127.13.238.129:50599 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 29 03:15:54 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:15:54 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:15:54.761535 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:54.756157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.129:50933)
0929 03:15:54.756386 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:54.756400 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:54.756487 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:54.756698 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:54.756701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:54.756853 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:54.756942 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:54.756948 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.757750 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.757753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:54.758626 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:54.758637 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.759153 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.759157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:54.759237 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:54.759657 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:54.759670 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:54.761408 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158}
W20260929 03:15:54.761657 23160 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:54.756255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:50933 (local address 127.13.238.190:44891)
0929 03:15:54.756376 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:54.756381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:54.756446 (+ 65us) server_negotiation.cc:407] Connection header received
0929 03:15:54.756566 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:54.756568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:54.756606 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:54.756668 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:54.757077 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.757652 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.758744 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.759039 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.759156 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:54.761538 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.129:50933: BlockingRecv error: recv got EOF from 127.13.238.129:50933 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 03:15:54.761766 23158 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:15:54.854609 23162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:54.854867 23162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:54.854915 23162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:54.859090 23162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:54.859164 23162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:54.859234 23162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:15:54.864714 23162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.130.keytab
--principal=oryx/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:41417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=37341
--tserver_master_addrs=127.13.238.190:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:54.865871 23162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:54.866986 23162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:54.876809 23167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:54.876801 23170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:54.876801 23168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:54.876890 23162 server_base.cc:1034] running on GCE node
I20260929 03:15:54.877483 23162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:54.878181 23162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:54.879388 23162 hybrid_clock.cc:648] HybridClock initialized: now 1790651754879367 us; error 32 us; skew 500 ppm
Sep 29 03:15:54 dist-test-slave-nbjg krb5kdc[22427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651754, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:54.882156 23162 init.cc:377] Logged in from keytab as oryx/127.13.238.130@KRBTEST.COM (short username oryx)
I20260929 03:15:54.883056 23162 webserver.cc:460] Webserver started at http://127.13.238.130:37341/ using document root <none> and password file <none>
I20260929 03:15:54.883596 23162 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:54.883666 23162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:54.887521 23162 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 03:15:54.889526 23177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:54.890810 23162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:15:54.890906 23162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "4081660706624c59931cd0c4f66068ad"
format_stamp: "Formatted at 2026-09-29 03:15:53 on dist-test-slave-nbjg"
server_key: "bb171152f4e75406258758d7f7d6f97e"
server_key_iv: "9856f63c91b19d0df0651f3b588e8a59"
server_key_version: "encryptionkey@0"
I20260929 03:15:54.891501 23162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:54.903503 23162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:54.906881 23162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:54.907037 23162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:54.907610 23162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:54.908437 23162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:54.908483 23162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:54.908524 23162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:54.908543 23162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:54.916674 23162 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:41417
I20260929 03:15:54.916702 23290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:41417 every 8 connection(s)
I20260929 03:15:54.917716 23162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:15:54 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:15:54 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:15:54.927709 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 23162
W20260929 03:15:54.928010 23160 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:54.919478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:44881 (local address 127.13.238.190:44891)
0929 03:15:54.919605 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:54.919609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:54.920700 (+ 1091us) server_negotiation.cc:407] Connection header received
0929 03:15:54.921508 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:54.921510 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:54.921562 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:54.921624 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:54.922947 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.923476 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.924558 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.924764 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.924833 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:54.927878 (+ 3045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.130:44881: BlockingRecv error: recv got EOF from 127.13.238.130:44881 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:15:54.928210 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:42753
--local_ip_for_outbound_sockets=127.13.238.131
--tserver_master_addrs=127.13.238.190:44891
--webserver_port=37575
--webserver_interface=127.13.238.131
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:34121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.238.131
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:15:54.928597 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:54.919838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.130:44881)
0929 03:15:54.920510 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:54.920552 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:54.921318 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:54.921850 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:54.921857 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:54.922287 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:54.922790 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:54.922804 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.923561 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.923563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:54.924464 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:54.924474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.924885 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.924888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:54.925027 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:54.925530 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:54.925547 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:54.927700 (+ 2153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":110,"threads_started":1}
W20260929 03:15:54.929073 23291 heartbeater.cc:644] Failed to heartbeat to 127.13.238.190:44891 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.238.190:44891: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Sep 29 03:15:54 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:15:54 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:15:54.934303 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:54.929487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.130:57183)
0929 03:15:54.929595 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:54.929608 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:54.929681 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:54.929914 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:54.929917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:54.930054 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:54.930143 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:54.930148 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.930869 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.930872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:54.931409 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:54.931417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.931822 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.931826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:54.931909 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:54.932416 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:54.932427 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:54.934144 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 03:15:54.934415 23160 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:54.929603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:57183 (local address 127.13.238.190:44891)
0929 03:15:54.929719 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:54.929723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:54.929736 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:15:54.929775 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:54.929777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:54.929813 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:54.929884 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:54.930265 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.930765 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.931517 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.931728 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.931811 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:54.934307 (+ 2496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.130:57183: BlockingRecv error: recv got EOF from 127.13.238.130:57183 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 29 03:15:54 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:15:54 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:15:54.939393 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:54.934948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.130:60907)
0929 03:15:54.935037 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:54.935050 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:54.935121 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:54.935343 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:54.935346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:54.935474 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:54.935539 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:54.935544 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.936226 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.936229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:54.936729 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:54.936736 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.937162 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.937166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:54.937237 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:54.937683 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:54.937693 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:54.939264 (+ 1571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 03:15:54.939497 23160 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:54.935044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:60907 (local address 127.13.238.190:44891)
0929 03:15:54.935163 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:54.935167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:54.935179 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:15:54.935213 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:54.935216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:54.935252 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:54.935313 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:54.935641 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.936131 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.936839 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:54.937050 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:54.937131 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:54.939397 (+ 2266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.130:60907: BlockingRecv error: recv got EOF from 127.13.238.130:60907 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 03:15:54.939729 23291 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:15:55.038164 23294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:15:55.038470 23294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:15:55.038527 23294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:15:55.042519 23294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:15:55.042580 23294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:15:55.042662 23294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:15:55.047976 23294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/krb5kdc/oryx_127.13.238.131.keytab
--principal=oryx/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:42753
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=37575
--tserver_master_addrs=127.13.238.190:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:15:55.049016 23294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:15:55.050056 23294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:15:55.059466 23302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:55.059460 23299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:15:55.059466 23300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:15:55.059756 23294 server_base.cc:1034] running on GCE node
I20260929 03:15:55.060137 23294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:15:55.060740 23294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:15:55.061897 23294 hybrid_clock.cc:648] HybridClock initialized: now 1790651755061871 us; error 38 us; skew 500 ppm
Sep 29 03:15:55 dist-test-slave-nbjg krb5kdc[22427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651755, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:15:55.064595 23294 init.cc:377] Logged in from keytab as oryx/127.13.238.131@KRBTEST.COM (short username oryx)
I20260929 03:15:55.065531 23294 webserver.cc:460] Webserver started at http://127.13.238.131:37575/ using document root <none> and password file <none>
I20260929 03:15:55.066071 23294 fs_manager.cc:359] Metadata directory not provided
I20260929 03:15:55.066147 23294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:15:55.070003 23294 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:15:55.072062 23309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:55.073282 23294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:15:55.073406 23294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "61b3440ce1e94ba59ba219aa9b8ec676"
format_stamp: "Formatted at 2026-09-29 03:15:54 on dist-test-slave-nbjg"
server_key: "c043d79fc04eef060b59317d17f0ed89"
server_key_iv: "97791778c4bf412687de54efc3653e0b"
server_key_version: "encryptionkey@0"
I20260929 03:15:55.074010 23294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:15:55.106617 23294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:15:55.109803 23294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:15:55.109997 23294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:15:55.110702 23294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:15:55.111557 23294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:15:55.111604 23294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:55.111660 23294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:15:55.111687 23294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:15:55.120242 23294 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:42753
I20260929 03:15:55.120278 23422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:42753 every 8 connection(s)
I20260929 03:15:55.121431 23294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:15:55.124771 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 23294
Sep 29 03:15:55 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:15:55 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:15:55.132784 23160 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:55.123267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:36493 (local address 127.13.238.190:44891)
0929 03:15:55.123436 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:55.123441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:55.124439 (+ 998us) server_negotiation.cc:407] Connection header received
0929 03:15:55.125524 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:55.125527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:55.125583 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:55.125673 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:55.126945 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:55.127463 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:55.128660 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:55.128952 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:55.129054 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:55.132661 (+ 3607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.131:36493: BlockingRecv error: recv got EOF from 127.13.238.131:36493 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 03:15:55.133535 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:55.123638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.131:36493)
0929 03:15:55.124291 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:55.124332 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:55.125284 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:55.125876 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:55.125887 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:55.126322 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:55.126807 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:55.126820 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:55.127575 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:55.127579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:55.128542 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:55.128552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:55.129146 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:55.129151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:55.129357 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:55.129882 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:55.129898 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:55.132556 (+ 2658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":120,"threads_started":1}
W20260929 03:15:55.134025 23423 heartbeater.cc:644] Failed to heartbeat to 127.13.238.190:44891 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.238.190:44891: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
I20260929 03:15:55.134407 23427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:55.125821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36182 (local address 127.13.238.190:44891)
0929 03:15:55.126044 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:55.126047 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:55.126057 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:15:55.126087 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:55.126089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:55.126124 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:55.126237 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:55.126974 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:55.127481 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:55.128297 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:55.128634 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:55.128718 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:55.130170 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:15:55.130177 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:15:55.130219 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:15:55.130250 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:15:55.132029 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:55.132522 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:55.132528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:55.132533 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:55.132668 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:15:55.132959 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:15:55.132962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:15:55.132964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:15:55.133425 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:15:55.133626 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 03:15:55.133977 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 03:15:55.134249 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":82,"threads_started":1}
Sep 29 03:15:55 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:15:55 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:15:55.139715 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:55.134475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.131:48959)
0929 03:15:55.134564 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:55.134578 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:55.134680 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:55.135470 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:55.135472 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:55.135633 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:55.135721 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:55.135727 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:55.136523 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:55.136527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:55.137165 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:55.137174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:55.137592 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:55.137596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:55.137678 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:55.138095 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:55.138106 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:55.139616 (+ 1510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 03:15:55.139829 23160 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:55.134640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:48959 (local address 127.13.238.190:44891)
0929 03:15:55.135275 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:55.135279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:55.135292 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:15:55.135323 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:55.135325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:55.135362 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:55.135437 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:55.135820 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:55.136441 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:55.137257 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:55.137467 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:55.137542 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:55.139724 (+ 2182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.131:48959: BlockingRecv error: recv got EOF from 127.13.238.131:48959 (error 108)
Metrics: {"server-negotiator.queue_time_us":560}
Sep 29 03:15:55 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:15:55 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:15:55.144320 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:55.140165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.131:37985)
0929 03:15:55.140255 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:55.140278 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:55.140352 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:55.140548 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:55.140550 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:55.140672 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:55.140737 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:55.140742 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:55.141392 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:55.141396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:55.141889 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:55.141896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:55.142319 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:55.142323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:55.142397 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:55.142730 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:55.142739 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:55.144231 (+ 1492us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 03:15:55.144444 23160 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:55.140238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:37985 (local address 127.13.238.190:44891)
0929 03:15:55.140383 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:55.140387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:55.140399 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:15:55.140430 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:55.140432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:55.140465 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:55.140524 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:55.140836 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:55.141301 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:55.141987 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:55.142188 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:55.142317 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:55.144344 (+ 2027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.131:37985: BlockingRecv error: recv got EOF from 127.13.238.131:37985 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260929 03:15:55.144506 23423 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 03:15:55 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:15:55 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:15:55.767695 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:55.762402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.129:52153)
0929 03:15:55.762649 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:55.762663 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:55.762784 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:55.763009 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:55.763011 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:55.763182 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:55.763278 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:55.763285 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:55.764052 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:55.764055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:55.764653 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:55.764663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:55.765050 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:55.765053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:55.765125 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:55.765570 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:55.765582 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:55.767571 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":70,"threads_started":1}
W20260929 03:15:55.768006 23429 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:55.762496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:52153 (local address 127.13.238.190:44891)
0929 03:15:55.762744 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:55.762748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:55.762761 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:15:55.762863 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:55.762865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:55.762908 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:55.762986 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:55.763389 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:55.763929 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:55.764746 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:55.764957 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:55.765023 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:55.767754 (+ 2731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.129:52153: BlockingRecv error: recv got EOF from 127.13.238.129:52153 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":94,"threads_started":1}
Sep 29 03:15:55 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:15:55 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:15:55.945966 23430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:55.940383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.130:43739)
0929 03:15:55.940636 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:55.940683 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:55.940783 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:55.941004 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:55.941006 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:55.941180 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:55.941295 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:55.941300 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:55.942099 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:55.942102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:55.942807 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:55.942822 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:55.943359 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:55.943363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:55.943445 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:55.944037 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:55.944052 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:55.945851 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":68,"threads_started":1}
W20260929 03:15:55.946148 23429 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:55.940507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:43739 (local address 127.13.238.190:44891)
0929 03:15:55.940724 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:55.940728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:55.940742 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:15:55.940863 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:55.940866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:55.940910 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:55.940978 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:55.941400 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:55.941976 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:55.942942 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:55.943246 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:55.943347 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:55.945999 (+ 2652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.130:43739: BlockingRecv error: recv got EOF from 127.13.238.130:43739 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
Sep 29 03:15:56 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:15:56 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:15:56.157316 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:56.145136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.131:56267)
0929 03:15:56.145414 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:56.145426 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:56.145529 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:56.145801 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:56.145803 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:56.145979 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:56.146099 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:56.146105 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:56.146916 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:56.146920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:56.147507 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:56.147515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:56.147882 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:56.147884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:56.147952 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:56.154963 (+ 7011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:56.154984 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:56.157159 (+ 2175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":72,"threads_started":1}
W20260929 03:15:56.157424 23429 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:56.145210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:56267 (local address 127.13.238.190:44891)
0929 03:15:56.145354 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:56.145359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:56.145493 (+ 134us) server_negotiation.cc:407] Connection header received
0929 03:15:56.145604 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:56.145606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:56.145655 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:56.145726 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:56.146253 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:56.146808 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:56.147595 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:56.147804 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:56.147873 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:56.157320 (+ 9447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.131:56267: BlockingRecv error: recv got EOF from 127.13.238.131:56267 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 03:15:56 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:15:56 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:15:56.773918 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:56.768519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.129:39111)
0929 03:15:56.768824 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:56.768837 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:56.768938 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:56.769224 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:56.769226 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:56.769400 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:56.769499 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:56.769505 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:56.770246 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:56.770250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:56.770799 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:56.770807 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:56.771323 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:56.771327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:56.771408 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:56.771968 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:56.771979 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:56.773778 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":116,"threads_started":1}
W20260929 03:15:56.774044 23433 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:56.768587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:39111 (local address 127.13.238.190:44891)
0929 03:15:56.768821 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:56.768825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:56.768905 (+ 80us) server_negotiation.cc:407] Connection header received
0929 03:15:56.769010 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:56.769012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:56.769057 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:56.769134 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:56.769594 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:56.770125 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:56.770919 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:56.771214 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:56.771308 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:56.773911 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.129:39111: BlockingRecv error: recv got EOF from 127.13.238.129:39111 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":67,"threads_started":1}
Sep 29 03:15:56 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:15:56 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:15:56.952225 23434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:56.946764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.130:50829)
0929 03:15:56.947057 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:56.947071 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:56.947197 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:56.947519 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:56.947522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:56.947716 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:56.947839 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:56.947849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:56.948744 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:56.948748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:56.949309 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:56.949318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:56.949735 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:56.949737 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:56.949798 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:56.950302 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:56.950313 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:56.952107 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":74,"threads_started":1}
W20260929 03:15:56.952353 23433 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:56.947164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:50829 (local address 127.13.238.190:44891)
0929 03:15:56.947301 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:56.947306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:56.947320 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:15:56.947361 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:56.947364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:56.947421 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:56.947498 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:56.947974 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:56.948635 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:56.949422 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:56.949641 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:56.949744 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:56.952246 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.130:50829: BlockingRecv error: recv got EOF from 127.13.238.130:50829 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 03:15:57 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:15:57 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:15:57.163753 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:57.158176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.131:39433)
0929 03:15:57.158530 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:57.158544 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:57.158663 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:57.158957 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:57.158960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:57.159147 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:57.159258 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:57.159267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:57.160006 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:57.160010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:57.160569 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:57.160577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:57.160997 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:57.161000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:57.161082 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:57.161573 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:57.161586 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:57.163579 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":94,"threads_started":1}
W20260929 03:15:57.163849 23433 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:57.158320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:39433 (local address 127.13.238.190:44891)
0929 03:15:57.158476 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:57.158482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:57.158625 (+ 143us) server_negotiation.cc:407] Connection header received
0929 03:15:57.158798 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:57.158802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:57.158852 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:57.158918 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:57.159374 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:57.159899 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:57.160679 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:57.160897 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:57.160983 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:57.163747 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.131:39433: BlockingRecv error: recv got EOF from 127.13.238.131:39433 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 03:15:57 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:15:57 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:15:57.779879 23436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:57.774818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.129:34065)
0929 03:15:57.775068 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:57.775081 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:57.775167 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:57.775383 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:57.775385 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:57.775559 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:57.775654 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:57.775660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:57.776401 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:57.776404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:57.776968 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:57.776975 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:57.777392 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:57.777394 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:57.777450 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:57.778031 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:57.778043 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:57.779760 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":69,"threads_started":1}
W20260929 03:15:57.780035 23437 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:57.774902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:34065 (local address 127.13.238.190:44891)
0929 03:15:57.775136 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:57.775139 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:57.775152 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:15:57.775239 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:57.775241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:57.775283 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:57.775360 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:57.775783 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:57.776319 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:57.777050 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:57.777307 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:57.777383 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:57.779895 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.129:34065: BlockingRecv error: recv got EOF from 127.13.238.129:34065 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":70,"threads_started":1}
Sep 29 03:15:57 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:15:57 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:15:57.958455 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:57.952955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.130:51827)
0929 03:15:57.953263 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:57.953278 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:57.953391 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:57.953630 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:57.953633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:57.953827 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:57.953958 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:57.953965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:57.954745 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:57.954749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:57.955436 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:57.955448 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:57.955946 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:57.955949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:57.956032 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:57.956520 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:57.956533 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:57.958321 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":100,"threads_started":1}
W20260929 03:15:57.958585 23437 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:57.953063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:51827 (local address 127.13.238.190:44891)
0929 03:15:57.953228 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:57.953233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:57.953344 (+ 111us) server_negotiation.cc:407] Connection header received
0929 03:15:57.953478 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:57.953481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:57.953527 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:57.953591 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:57.954067 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:57.954633 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:57.955549 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:57.955818 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:57.955925 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:57.958464 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.130:51827: BlockingRecv error: recv got EOF from 127.13.238.130:51827 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 03:15:58 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:15:58 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:15:58.169574 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:58.164614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.131:48569)
0929 03:15:58.164916 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:58.164929 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:58.165031 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:58.165254 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:58.165256 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:58.165438 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:58.165533 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:58.165538 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:58.166288 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:58.166290 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:58.166849 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:58.166857 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:58.167237 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:58.167239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:58.167308 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:58.167785 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:58.167797 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:58.169447 (+ 1650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":81,"threads_started":1}
W20260929 03:15:58.169737 23437 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:58.164708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:48569 (local address 127.13.238.190:44891)
0929 03:15:58.164848 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:58.164852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:58.164995 (+ 143us) server_negotiation.cc:407] Connection header received
0929 03:15:58.165104 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:58.165106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:58.165159 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:58.165227 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:58.165616 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:58.166139 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:58.166937 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:58.167146 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:58.167231 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:58.169610 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.131:48569: BlockingRecv error: recv got EOF from 127.13.238.131:48569 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 03:15:58 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:15:58 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:15:58.785684 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:58.780706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.129:56005)
0929 03:15:58.780948 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:58.780961 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:58.781058 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:58.781281 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:58.781283 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:58.781450 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:58.781544 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:58.781550 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:58.782304 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:58.782307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:58.782873 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:58.782883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:58.783261 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:58.783263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:58.783324 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:58.783787 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:58.783799 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:58.785562 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
W20260929 03:15:58.785825 23441 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:58.780705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:56005 (local address 127.13.238.190:44891)
0929 03:15:58.780952 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:58.780956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:58.781021 (+ 65us) server_negotiation.cc:407] Connection header received
0929 03:15:58.781141 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:58.781143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:58.781186 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:58.781253 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:58.781628 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:58.782150 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:58.782964 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:58.783177 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:58.783248 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:58.785689 (+ 2441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.129:56005: BlockingRecv error: recv got EOF from 127.13.238.129:56005 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
Sep 29 03:15:58 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:15:58 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:15:58.964428 23442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:58.959242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.130:45101)
0929 03:15:58.959518 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:58.959531 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:58.959647 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:58.959873 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:58.959875 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:58.960043 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:58.960144 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:58.960149 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:58.960856 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:58.960858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:58.961406 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:58.961413 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:58.961888 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:58.961892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:58.961959 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:58.962481 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:58.962493 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:58.964279 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":88,"threads_started":1}
W20260929 03:15:58.964516 23441 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:58.959320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:45101 (local address 127.13.238.190:44891)
0929 03:15:58.959471 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:58.959476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:58.959596 (+ 120us) server_negotiation.cc:407] Connection header received
0929 03:15:58.959720 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:58.959723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:58.959768 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:58.959835 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:58.960230 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:58.960761 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:58.961528 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:58.961778 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:58.961870 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:58.964416 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.130:45101: BlockingRecv error: recv got EOF from 127.13.238.130:45101 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 03:15:59 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:15:59 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:15:59.175688 23443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:59.170289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.131:36545)
0929 03:15:59.170585 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:59.170598 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:59.170719 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:59.170965 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:59.170969 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:59.171173 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:59.171268 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:59.171274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:59.172051 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:59.172056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:59.172620 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:59.172629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:59.173047 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:59.173052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:59.173146 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:59.173667 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:59.173682 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:59.175538 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":83,"threads_started":1}
W20260929 03:15:59.175804 23441 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:59.170382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:36545 (local address 127.13.238.190:44891)
0929 03:15:59.170526 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:59.170531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:59.170677 (+ 146us) server_negotiation.cc:407] Connection header received
0929 03:15:59.170811 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:59.170814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:59.170859 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:59.170923 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:59.171385 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:59.171943 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:59.172735 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:59.172948 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:59.173032 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:59.175640 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.131:36545: BlockingRecv error: recv got EOF from 127.13.238.131:36545 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 03:15:59 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:15:59 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:15:59.791873 23444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:59.786488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.129:44577)
0929 03:15:59.786823 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:59.786837 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:59.786943 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:59.787182 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:59.787185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:59.787376 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:59.787466 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:59.787473 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:59.788203 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:59.788207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:59.788764 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:59.788781 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:59.789205 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:59.789209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:59.789292 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:59.789765 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:59.789783 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:59.791700 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":115,"threads_started":1}
W20260929 03:15:59.792045 23445 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:59.786533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:44577 (local address 127.13.238.190:44891)
0929 03:15:59.786775 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:59.786779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:59.786902 (+ 123us) server_negotiation.cc:407] Connection header received
0929 03:15:59.787028 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:59.787031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:59.787077 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:59.787149 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:59.787576 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:59.788104 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:59.788881 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:59.789106 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:59.789190 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:59.791918 (+ 2728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.129:44577: BlockingRecv error: recv got EOF from 127.13.238.129:44577 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":74,"threads_started":1}
Sep 29 03:15:59 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:15:59 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:15:59.970599 23446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:15:59.965273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.130:43681)
0929 03:15:59.965558 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 03:15:59.965612 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 03:15:59.965708 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:15:59.965915 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:15:59.965916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:15:59.966092 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:15:59.966184 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:59.966190 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:59.966972 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:59.966975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:59.967584 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:15:59.967594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:15:59.968042 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:15:59.968045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:15:59.968110 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:59.968555 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:15:59.968566 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:15:59.970442 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":89,"threads_started":1}
W20260929 03:15:59.970748 23445 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:15:59.965450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:43681 (local address 127.13.238.190:44891)
0929 03:15:59.965603 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 03:15:59.965606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:15:59.965664 (+ 58us) server_negotiation.cc:407] Connection header received
0929 03:15:59.965777 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:15:59.965779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:15:59.965821 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:15:59.965890 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:15:59.966308 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:59.966850 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:59.967723 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:15:59.967941 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:15:59.968013 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:15:59.970623 (+ 2610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.130:43681: BlockingRecv error: recv got EOF from 127.13.238.130:43681 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 03:16:00 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:00 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:00.182160 23447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:00.176421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.131:53815)
0929 03:16:00.176710 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:00.176756 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:00.176865 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:00.177104 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:00.177107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:00.177309 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:00.177402 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:00.177408 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:00.178247 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:00.178251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:00.178890 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:00.178900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:00.179328 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:00.179332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:00.179416 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:00.179886 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:00.179898 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:00.182007 (+ 2109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":86,"threads_started":1}
W20260929 03:16:00.182294 23445 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:00.176534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:53815 (local address 127.13.238.190:44891)
0929 03:16:00.176711 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:00.176716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:00.176822 (+ 106us) server_negotiation.cc:407] Connection header received
0929 03:16:00.176951 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:00.176954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:00.177002 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:00.177067 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:00.177541 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:00.178111 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:00.179000 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:00.179220 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:00.179314 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:00.182157 (+ 2843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.131:53815: BlockingRecv error: recv got EOF from 127.13.238.131:53815 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 29 03:16:00 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:00 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:00.798153 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:00.792907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.129:56919)
0929 03:16:00.793171 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:00.793185 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:00.793279 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:00.793518 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:00.793520 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:00.793724 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:00.793824 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:00.793831 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:00.794560 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:00.794563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:00.795218 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:00.795229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:00.795680 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:00.795683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:00.795745 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:00.796289 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:00.796303 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:00.798029 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":73,"threads_started":1}
W20260929 03:16:00.798342 23449 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:00.792988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:56919 (local address 127.13.238.190:44891)
0929 03:16:00.793212 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:00.793222 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:00.793235 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:16:00.793358 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:00.793361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:00.793409 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:00.793482 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:00.793915 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:00.794469 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:00.795317 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:00.795560 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:00.795634 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:00.798171 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.129:56919: BlockingRecv error: recv got EOF from 127.13.238.129:56919 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":65,"threads_started":1}
Sep 29 03:16:00 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:00 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:00.976677 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:00.971542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.130:48865)
0929 03:16:00.971817 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:00.971830 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:00.971930 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:00.972152 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:00.972154 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:00.972331 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:00.972425 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:00.972430 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:00.973154 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:00.973158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:00.973739 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:00.973748 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:00.974168 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:00.974171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:00.974287 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:00.974776 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:00.974789 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:00.976556 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":71,"threads_started":1}
W20260929 03:16:00.976851 23449 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:00.971619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:48865 (local address 127.13.238.190:44891)
0929 03:16:00.971751 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:00.971754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:00.971890 (+ 136us) server_negotiation.cc:407] Connection header received
0929 03:16:00.972020 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:00.972023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:00.972068 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:00.972126 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:00.972526 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:00.973047 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:00.973842 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:00.974058 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:00.974130 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:00.976728 (+ 2598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.130:48865: BlockingRecv error: recv got EOF from 127.13.238.130:48865 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 03:16:01 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:01 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:01.192293 23451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:01.182998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.131:60659)
0929 03:16:01.183282 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:01.183297 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:01.183402 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:01.183635 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:01.183646 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:01.183845 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:01.183936 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:01.183942 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:01.184746 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:01.184749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:01.185300 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:01.185307 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:01.185721 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:01.185725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:01.185806 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:01.190219 (+ 4413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:01.190232 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:01.192148 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":96,"threads_started":1}
W20260929 03:16:01.192454 23449 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:01.183107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:60659 (local address 127.13.238.190:44891)
0929 03:16:01.183250 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:01.183255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:01.183359 (+ 104us) server_negotiation.cc:407] Connection header received
0929 03:16:01.183487 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:01.183490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:01.183536 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:01.183601 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:01.184048 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:01.184593 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:01.185410 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:01.185622 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:01.185709 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:01.192330 (+ 6621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.131:60659: BlockingRecv error: recv got EOF from 127.13.238.131:60659 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 03:16:01 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:01 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:01.804626 23452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:01.799089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.129:43071)
0929 03:16:01.799398 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:01.799411 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:01.799518 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:01.799809 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:01.799812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:01.799991 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:01.800084 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:01.800090 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:01.800894 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:01.800898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:01.801477 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:01.801487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:01.801957 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:01.801960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:01.802027 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:01.802503 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:01.802514 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:01.804498 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":70,"threads_started":1}
W20260929 03:16:01.804862 23453 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:01.799192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:43071 (local address 127.13.238.190:44891)
0929 03:16:01.799434 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:01.799437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:01.799454 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:16:01.799606 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:01.799608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:01.799653 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:01.799736 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:01.800233 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:01.800785 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:01.801596 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:01.801848 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:01.801930 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:01.804719 (+ 2789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.129:43071: BlockingRecv error: recv got EOF from 127.13.238.129:43071 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":69,"threads_started":1}
Sep 29 03:16:01 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:01 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:01.983179 23454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:01.977595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.130:60957)
0929 03:16:01.977923 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:01.977937 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:01.978056 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:01.978356 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:01.978359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:01.978562 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:01.978658 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:01.978665 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:01.979433 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:01.979437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:01.980024 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:01.980034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:01.980482 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:01.980486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:01.980567 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:01.981074 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:01.981088 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:01.983052 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":82,"threads_started":1}
W20260929 03:16:01.983367 23453 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:01.977688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:60957 (local address 127.13.238.190:44891)
0929 03:16:01.977850 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:01.977856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:01.978013 (+ 157us) server_negotiation.cc:407] Connection header received
0929 03:16:01.978147 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:01.978151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:01.978242 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:01.978317 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:01.978783 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:01.979326 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:01.980144 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:01.980378 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:01.980464 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:01.983235 (+ 2771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.130:60957: BlockingRecv error: recv got EOF from 127.13.238.130:60957 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 03:16:02 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:02 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:02.199086 23455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:02.193224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.131:45745)
0929 03:16:02.193524 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:02.193537 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:02.193695 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:02.193923 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:02.193925 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:02.194107 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:02.194275 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:02.194283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:02.195197 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:02.195200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:02.195938 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:02.195949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:02.196410 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:02.196415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:02.196486 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:02.197032 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:02.197044 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:02.198934 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":71,"threads_started":1}
W20260929 03:16:02.199204 23453 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:02.193314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:45745 (local address 127.13.238.190:44891)
0929 03:16:02.193454 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:02.193457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:02.193653 (+ 196us) server_negotiation.cc:407] Connection header received
0929 03:16:02.193773 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:02.193775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:02.193823 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:02.193898 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:02.194407 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:02.195077 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:02.196055 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:02.196283 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:02.196380 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:02.199086 (+ 2706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.131:45745: BlockingRecv error: recv got EOF from 127.13.238.131:45745 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 03:16:02 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:02 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:02.810671 23456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:02.805401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.129:55861)
0929 03:16:02.805667 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:02.805684 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:02.805790 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:02.806032 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:02.806035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:02.806271 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:02.806367 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:02.806374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:02.807119 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:02.807130 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:02.807687 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:02.807695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:02.808129 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:02.808132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:02.808215 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:02.808687 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:02.808697 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:02.810545 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":76,"threads_started":1}
W20260929 03:16:02.810873 23457 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:02.805510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:55861 (local address 127.13.238.190:44891)
0929 03:16:02.805780 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:02.805785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:02.805798 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:16:02.805871 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:02.805873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:02.805921 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:02.805992 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:02.806487 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:02.807018 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:02.807800 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:02.808025 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:02.808110 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:02.810740 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.129:55861: BlockingRecv error: recv got EOF from 127.13.238.129:55861 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":69,"threads_started":1}
Sep 29 03:16:02 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:02 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:02.990015 23458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:02.984005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.130:47211)
0929 03:16:02.984293 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:02.984305 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:02.984410 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:02.984669 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:02.984672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:02.984851 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:02.984964 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:02.984970 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:02.985957 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:02.985960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:02.986571 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:02.986582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:02.987131 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:02.987134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:02.987201 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:02.987751 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:02.987763 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:02.989865 (+ 2102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":79,"threads_started":1}
W20260929 03:16:02.990262 23457 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:02.984066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:47211 (local address 127.13.238.190:44891)
0929 03:16:02.984204 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:02.984208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:02.984375 (+ 167us) server_negotiation.cc:407] Connection header received
0929 03:16:02.984477 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:02.984479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:02.984528 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:02.984594 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:02.985069 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:02.985819 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:02.986736 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:02.986991 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:02.987079 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:02.990091 (+ 3012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.130:47211: BlockingRecv error: recv got EOF from 127.13.238.130:47211 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 03:16:03 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:03 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:16:03.207041 23457 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:03.200103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:56807 (local address 127.13.238.190:44891)
0929 03:16:03.200276 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:03.200280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:03.200450 (+ 170us) server_negotiation.cc:407] Connection header received
0929 03:16:03.200602 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:03.200605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:03.200661 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:03.200751 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:03.201278 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:03.202060 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:03.203332 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:03.203650 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:03.203752 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:03.206900 (+ 3148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.131:56807: BlockingRecv error: recv got EOF from 127.13.238.131:56807 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:16:03.207402 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:03.200009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.131:56807)
0929 03:16:03.200348 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:03.200364 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:03.200499 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:03.200791 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:03.200794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:03.201022 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:03.201157 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:03.201164 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:03.202176 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:03.202181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:03.203200 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:03.203213 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:03.203794 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:03.203798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:03.203887 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:03.204479 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:03.204495 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:03.207267 (+ 2772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":75,"threads_started":1}
Sep 29 03:16:03 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:03 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:03.817368 23460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:03.811563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.129:57861)
0929 03:16:03.811865 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:03.811878 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:03.811984 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:03.812289 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:03.812292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:03.812499 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:03.812605 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:03.812612 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:03.813446 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:03.813449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:03.814052 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:03.814062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:03.814637 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:03.814641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:03.814756 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:03.815305 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:03.815321 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:03.817228 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":134,"threads_started":1}
W20260929 03:16:03.817500 23461 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:03.811652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:57861 (local address 127.13.238.190:44891)
0929 03:16:03.811888 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:03.811892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:03.811938 (+ 46us) server_negotiation.cc:407] Connection header received
0929 03:16:03.812067 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:03.812070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:03.812118 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:03.812211 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:03.812761 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:03.813320 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:03.814188 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:03.814519 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:03.814618 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:03.817379 (+ 2761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.129:57861: BlockingRecv error: recv got EOF from 127.13.238.129:57861 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":61,"threads_started":1}
Sep 29 03:16:03 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:03 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:03.996654 23462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:03.991127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.130:39081)
0929 03:16:03.991415 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:03.991429 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:03.991551 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:03.991850 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:03.991854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:03.992046 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:03.992160 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:03.992177 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:03.992968 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:03.992971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:03.993534 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:03.993543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:03.994001 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:03.994003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:03.994064 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:03.994617 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:03.994634 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:03.996533 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":82,"threads_started":1}
W20260929 03:16:03.996886 23461 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:03.991209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:39081 (local address 127.13.238.190:44891)
0929 03:16:03.991358 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:03.991363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:03.991509 (+ 146us) server_negotiation.cc:407] Connection header received
0929 03:16:03.991660 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:03.991663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:03.991721 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:03.991790 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:03.992264 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:03.992870 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:03.993647 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:03.993908 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:03.993990 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:03.996768 (+ 2778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.130:39081: BlockingRecv error: recv got EOF from 127.13.238.130:39081 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 29 03:16:04 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:04 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:04.213652 23463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:04.208176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.131:56175)
0929 03:16:04.208480 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:04.208492 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:04.208602 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:04.208894 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:04.208897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:04.209093 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:04.209184 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:04.209190 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:04.209940 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:04.209944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:04.210578 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:04.210587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:04.211020 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:04.211024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:04.211100 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:04.211602 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:04.211615 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:04.213530 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":114,"threads_started":1}
W20260929 03:16:04.213814 23461 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:04.208261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:56175 (local address 127.13.238.190:44891)
0929 03:16:04.208402 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:04.208407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:04.208561 (+ 154us) server_negotiation.cc:407] Connection header received
0929 03:16:04.208690 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:04.208693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:04.208783 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:04.208853 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:04.209316 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:04.209839 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:04.210687 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:04.210920 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:04.211009 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:04.213683 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.131:56175: BlockingRecv error: recv got EOF from 127.13.238.131:56175 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 03:16:04 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:04 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:04.823606 23464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:04.818262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.129:43125)
0929 03:16:04.818528 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:04.818541 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:04.818648 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:04.818883 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:04.818885 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:04.819066 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:04.819163 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:04.819169 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:04.819982 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:04.819985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:04.820553 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:04.820561 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:04.821034 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:04.821037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:04.821104 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:04.821584 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:04.821596 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:04.823479 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":73,"threads_started":1}
W20260929 03:16:04.823853 23465 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:04.818305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:43125 (local address 127.13.238.190:44891)
0929 03:16:04.818563 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:04.818567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:04.818597 (+ 30us) server_negotiation.cc:407] Connection header received
0929 03:16:04.818730 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:04.818732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:04.818776 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:04.818845 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:04.819274 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:04.819876 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:04.820678 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:04.820923 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:04.821009 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:04.823721 (+ 2712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.129:43125: BlockingRecv error: recv got EOF from 127.13.238.129:43125 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":71,"threads_started":1}
Sep 29 03:16:04 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:04 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:05.002868 23466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:04.997417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.130:58201)
0929 03:16:04.997774 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:04.997788 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:04.997909 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:04.998163 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:04.998166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:04.998391 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:04.998488 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:04.998494 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:04.999248 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:04.999251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:04.999821 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:04.999831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:05.000342 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:05.000346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:05.000427 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:05.000906 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:05.000917 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:05.002734 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":104,"threads_started":1}
W20260929 03:16:05.003000 23465 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:04.997571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:58201 (local address 127.13.238.190:44891)
0929 03:16:04.997764 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:04.997769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:04.997866 (+ 97us) server_negotiation.cc:407] Connection header received
0929 03:16:04.998001 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:04.998005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:04.998055 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:04.998121 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:04.998611 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:04.999130 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:04.999943 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:05.000233 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:05.000326 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:05.002879 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.130:58201: BlockingRecv error: recv got EOF from 127.13.238.130:58201 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 03:16:05 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:05 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:05.220237 23467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:05.214535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.131:44831)
0929 03:16:05.214798 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:05.214814 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:05.214917 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:05.215143 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:05.215145 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:05.215325 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:05.215444 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:05.215450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:05.216246 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:05.216250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:05.216824 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:05.216835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:05.217320 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:05.217323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:05.217399 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:05.218006 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:05.218021 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:05.220060 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":73,"threads_started":1}
W20260929 03:16:05.220300 23465 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:05.214696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:44831 (local address 127.13.238.190:44891)
0929 03:16:05.214825 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:05.214829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:05.214870 (+ 41us) server_negotiation.cc:407] Connection header received
0929 03:16:05.214988 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:05.214990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:05.215036 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:05.215112 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:05.215549 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:05.216103 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:05.216947 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:05.217210 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:05.217312 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:05.220195 (+ 2883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.131:44831: BlockingRecv error: recv got EOF from 127.13.238.131:44831 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 03:16:05 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:05 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:05.829558 23468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:05.824451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.129:41471)
0929 03:16:05.824706 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:05.824718 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:05.824804 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:05.825032 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:05.825034 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:05.825215 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:05.825311 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:05.825317 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:05.826050 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:05.826052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:05.826645 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:05.826653 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:05.827025 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:05.827027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:05.827086 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:05.827603 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:05.827617 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:05.829434 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
W20260929 03:16:05.829738 23469 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:05.824516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:41471 (local address 127.13.238.190:44891)
0929 03:16:05.824763 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:05.824766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:05.824779 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:16:05.824883 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:05.824885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:05.824929 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:05.825004 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:05.825400 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:05.825955 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:05.826728 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:05.826944 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:05.827014 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:05.829599 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.129:41471: BlockingRecv error: recv got EOF from 127.13.238.129:41471 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":82,"threads_started":1}
Sep 29 03:16:06 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:06 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:06.009308 23470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:06.003690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.130:49745)
0929 03:16:06.003974 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:06.003987 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:06.004100 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:06.004332 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:06.004334 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:06.004511 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:06.004621 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:06.004628 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:06.005414 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:06.005417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:06.005992 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:06.006001 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:06.006544 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:06.006548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:06.006626 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:06.007113 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:06.007126 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:06.009146 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":97,"threads_started":1}
W20260929 03:16:06.009446 23469 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:06.003771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:49745 (local address 127.13.238.190:44891)
0929 03:16:06.003934 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:06.003938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:06.004075 (+ 137us) server_negotiation.cc:407] Connection header received
0929 03:16:06.004176 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:06.004178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:06.004229 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:06.004299 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:06.004755 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:06.005311 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:06.006122 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:06.006404 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:06.006498 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:06.009336 (+ 2838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.130:49745: BlockingRecv error: recv got EOF from 127.13.238.130:49745 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 03:16:06 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:06 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:06.230507 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:06.221081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.131:50271)
0929 03:16:06.221395 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:06.221410 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:06.221532 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:06.221795 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:06.221798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:06.221989 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:06.222099 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:06.222106 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:06.222900 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:06.222903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:06.223466 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:06.223475 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:06.223916 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:06.223920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:06.224004 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:06.228201 (+ 4197us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:06.228220 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:06.230349 (+ 2129us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":109,"threads_started":1}
W20260929 03:16:06.230604 23469 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:06.221224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:50271 (local address 127.13.238.190:44891)
0929 03:16:06.221396 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:06.221401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:06.221483 (+ 82us) server_negotiation.cc:407] Connection header received
0929 03:16:06.221640 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:06.221644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:06.221693 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:06.221756 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:06.222244 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:06.222789 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:06.223584 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:06.223811 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:06.223900 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:06.230498 (+ 6598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.131:50271: BlockingRecv error: recv got EOF from 127.13.238.131:50271 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 29 03:16:06 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:06 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:06.836099 23472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:06.830460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.129:59555)
0929 03:16:06.830753 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:06.830769 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:06.830873 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:06.831151 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:06.831154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:06.831335 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:06.831442 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:06.831448 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:06.832285 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:06.832288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:06.832849 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:06.832859 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:06.833454 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:06.833459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:06.833587 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:06.834140 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:06.834155 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:06.835982 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":98,"threads_started":1}
W20260929 03:16:06.836239 23473 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:06.830528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:59555 (local address 127.13.238.190:44891)
0929 03:16:06.830766 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:06.830769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:06.830819 (+ 50us) server_negotiation.cc:407] Connection header received
0929 03:16:06.830941 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:06.830943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:06.830985 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:06.831065 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:06.831584 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:06.832165 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:06.832988 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:06.833322 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:06.833419 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:06.836107 (+ 2688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.129:59555: BlockingRecv error: recv got EOF from 127.13.238.129:59555 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":70,"threads_started":1}
Sep 29 03:16:07 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:07 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:07.015144 23474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:07.010181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.130:53691)
0929 03:16:07.010435 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:07.010448 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:07.010549 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:07.010765 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:07.010767 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:07.010936 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:07.011028 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:07.011032 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:07.011729 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:07.011731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:07.012289 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:07.012296 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:07.012736 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:07.012738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:07.012792 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:07.013230 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:07.013242 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:07.015026 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
W20260929 03:16:07.015298 23473 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:07.010252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:53691 (local address 127.13.238.190:44891)
0929 03:16:07.010399 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:07.010404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:07.010536 (+ 132us) server_negotiation.cc:407] Connection header received
0929 03:16:07.010615 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:07.010618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:07.010673 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:07.010743 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:07.011111 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:07.011653 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:07.012413 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:07.012647 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:07.012729 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:07.015182 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.130:53691: BlockingRecv error: recv got EOF from 127.13.238.130:53691 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 03:16:07 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:07 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:07.236479 23475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:07.231333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.131:35069)
0929 03:16:07.231654 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:07.231670 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:07.231776 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:07.232021 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:07.232024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:07.232226 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:07.232320 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:07.232327 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:07.233060 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:07.233064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:07.233628 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:07.233636 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:07.234058 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:07.234062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:07.234142 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:07.234635 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:07.234647 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:07.236330 (+ 1683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":99,"threads_started":1}
W20260929 03:16:07.236577 23473 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:07.231381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:35069 (local address 127.13.238.190:44891)
0929 03:16:07.231515 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:07.231520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:07.231736 (+ 216us) server_negotiation.cc:407] Connection header received
0929 03:16:07.231865 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:07.231868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:07.231921 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:07.231987 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:07.232430 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:07.232959 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:07.233735 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:07.233962 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:07.234054 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:07.236471 (+ 2417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.131:35069: BlockingRecv error: recv got EOF from 127.13.238.131:35069 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 03:16:07 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:07 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:07.841934 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:07.836935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.129:47169)
0929 03:16:07.837186 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:07.837199 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:07.837286 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:07.837486 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:07.837488 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:07.837671 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:07.837766 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:07.837771 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:07.838579 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:07.838581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:07.839147 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:07.839154 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:07.839551 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:07.839553 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:07.839610 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:07.840043 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:07.840055 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:07.841804 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":71,"threads_started":1}
W20260929 03:16:07.842062 23477 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:07.837017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:47169 (local address 127.13.238.190:44891)
0929 03:16:07.837270 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:07.837274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:07.837287 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:16:07.837354 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:07.837357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:07.837401 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:07.837478 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:07.837901 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:07.838498 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:07.839243 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:07.839466 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:07.839538 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:07.841943 (+ 2405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.129:47169: BlockingRecv error: recv got EOF from 127.13.238.129:47169 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":69,"threads_started":1}
Sep 29 03:16:08 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:08 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:08.021274 23478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:08.015927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.130:38763)
0929 03:16:08.016193 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:08.016206 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:08.016307 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:08.016645 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:08.016648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:08.016846 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:08.016941 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:08.016947 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:08.017715 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:08.017717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:08.018316 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:08.018325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:08.018792 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:08.018795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:08.018855 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:08.019314 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:08.019326 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:08.021106 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":85,"threads_started":1}
W20260929 03:16:08.021363 23477 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:08.016070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:38763 (local address 127.13.238.190:44891)
0929 03:16:08.016204 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:08.016207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:08.016267 (+ 60us) server_negotiation.cc:407] Connection header received
0929 03:16:08.016425 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:08.016428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:08.016479 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:08.016607 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:08.017047 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:08.017620 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:08.018418 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:08.018677 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:08.018756 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:08.021256 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.130:38763: BlockingRecv error: recv got EOF from 127.13.238.130:38763 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 03:16:08 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:08 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:08.242182 23479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:08.237236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.131:49855)
0929 03:16:08.237488 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:08.237501 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:08.237639 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:08.237845 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:08.237847 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:08.238029 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:08.238125 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:08.238130 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:08.238880 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:08.238883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:08.239464 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:08.239474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:08.239872 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:08.239874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:08.239933 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:08.240389 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:08.240401 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:08.242064 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
W20260929 03:16:08.242349 23477 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:08.237311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:49855 (local address 127.13.238.190:44891)
0929 03:16:08.237432 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:08.237435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:08.237587 (+ 152us) server_negotiation.cc:407] Connection header received
0929 03:16:08.237716 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:08.237719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:08.237768 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:08.237831 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:08.238241 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:08.238757 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:08.239570 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:08.239790 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:08.239862 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:08.242228 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.131:49855: BlockingRecv error: recv got EOF from 127.13.238.131:49855 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 03:16:08 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:08 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:08.848094 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:08.842692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.129:58029)
0929 03:16:08.842994 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:08.843012 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:08.843127 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:08.843402 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:08.843404 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:08.843586 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:08.843694 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:08.843700 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:08.844523 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:08.844527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:08.845171 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:08.845181 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:08.845636 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:08.845639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:08.845724 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:08.846225 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:08.846242 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:08.847964 (+ 1722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":90,"threads_started":1}
W20260929 03:16:08.848212 23481 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:08.842788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:58029 (local address 127.13.238.190:44891)
0929 03:16:08.843108 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:08.843112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:08.843126 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:16:08.843207 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:08.843210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:08.843259 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:08.843332 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:08.843843 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:08.844401 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:08.845301 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:08.845539 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:08.845619 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:08.848096 (+ 2477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.129:58029: BlockingRecv error: recv got EOF from 127.13.238.129:58029 (error 108)
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":73,"threads_started":1}
Sep 29 03:16:09 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:09 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:09.028102 23482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:09.022226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.130:37817)
0929 03:16:09.022487 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:09.022499 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:09.022606 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:09.022928 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:09.022930 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:09.023105 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:09.023201 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:09.023206 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:09.024040 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:09.024043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:09.024638 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:09.024648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:09.025179 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:09.025184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:09.025270 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:09.025888 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:09.025905 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:09.027978 (+ 2073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
W20260929 03:16:09.028276 23481 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:09.022298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:37817 (local address 127.13.238.190:44891)
0929 03:16:09.022451 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:09.022455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:09.022571 (+ 116us) server_negotiation.cc:407] Connection header received
0929 03:16:09.022774 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:09.022777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:09.022828 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:09.022902 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:09.023334 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:09.023907 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:09.024798 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:09.025044 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:09.025135 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:09.028143 (+ 3008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.130:37817: BlockingRecv error: recv got EOF from 127.13.238.130:37817 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 03:16:09 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:09 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.131@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:09.248596 23483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:09.243041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.131:40293)
0929 03:16:09.243325 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:09.243339 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:09.243444 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:09.243716 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:09.243718 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:09.243897 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:09.243996 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:09.244001 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:09.244777 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:09.244779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:09.245354 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:09.245362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:09.245755 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:09.245758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:09.245820 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:09.246365 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:09.246378 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:09.248447 (+ 2069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":96,"threads_started":1}
W20260929 03:16:09.248687 23481 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:09.243183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:40293 (local address 127.13.238.190:44891)
0929 03:16:09.243314 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:09.243318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:09.243434 (+ 116us) server_negotiation.cc:407] Connection header received
0929 03:16:09.243514 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:09.243517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:09.243571 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:09.243645 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:09.244135 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:09.244679 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:09.245446 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:09.245663 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:09.245743 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:09.248583 (+ 2840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.131:40293: BlockingRecv error: recv got EOF from 127.13.238.131:40293 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 03:16:09 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:09 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.129@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:09.854331 23484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:09.848960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.129:47313)
0929 03:16:09.849267 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:09.849281 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:09.849374 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:09.849646 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:09.849648 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:09.849830 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:09.849925 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:09.849931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:09.850669 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:09.850672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:09.851270 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:09.851280 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:09.851736 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:09.851740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:09.851817 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:09.852307 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:09.852320 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:09.854155 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":74,"threads_started":1}
W20260929 03:16:09.854452 23485 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:09.849093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:47313 (local address 127.13.238.190:44891)
0929 03:16:09.849336 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:09.849339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:09.849353 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:16:09.849451 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:09.849453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:09.849493 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:09.849567 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:09.850013 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:09.850558 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:09.851378 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:09.851605 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:09.851692 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:09.854295 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.129:47313: BlockingRecv error: recv got EOF from 127.13.238.129:47313 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":80,"threads_started":1}
Sep 29 03:16:10 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:16:10 dist-test-slave-nbjg krb5kdc[22427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.238.130@KRBTEST.COM for oryx/127.13.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:16:10.034373 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:10.029067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:44891 (local address 127.13.238.130:48853)
0929 03:16:10.029359 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:10.029373 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:10.029483 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:10.029732 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:10.029735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:10.029955 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:10.030067 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:10.030074 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:10.030950 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:10.030953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:10.031524 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:10.031534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:10.031977 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:10.031980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:10.032059 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:10.032519 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:10.032531 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:10.034170 (+ 1639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.238.190:44891: Server oryx/127.13.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":75,"threads_started":1}
W20260929 03:16:10.034456 23485 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:10.029156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:48853 (local address 127.13.238.190:44891)
0929 03:16:10.029299 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:10.029304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:10.029464 (+ 160us) server_negotiation.cc:407] Connection header received
0929 03:16:10.029581 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:10.029584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:10.029630 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:10.029694 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:10.030180 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:10.030848 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:10.031637 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:10.031876 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:10.031961 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:10.034349 (+ 2388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.238.130:48853: BlockingRecv error: recv got EOF from 127.13.238.130:48853 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:16:10.133649 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 23010
I20260929 03:16:10.139029 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 23162
I20260929 03:16:10.143730 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 23294
I20260929 03:16:10.148061 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 22940
2026-09-29T03:16:10Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17888 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T03:16:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:16:10Z Disabled control of system clock
I20260929 03:16:10.176267 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:35893
--webserver_interface=127.13.238.190
--webserver_port=0
--builtin_ntp_servers=127.13.238.148:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:35893
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 03:16:10.287225 23490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:16:10.287498 23490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:16:10.287532 23490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:16:10.291471 23490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:16:10.291522 23490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:16:10.291535 23490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:16:10.291550 23490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:16:10.291563 23490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:16:10.296273 23490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:35893
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:35893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:16:10.297330 23490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:16:10.298729 23490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:16:10.307327 23498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:16:10.307343 23495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:16:10.307394 23490 server_base.cc:1034] running on GCE node
W20260929 03:16:10.307334 23496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:16:10.308166 23490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:16:10.309305 23490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:16:10.310461 23490 hybrid_clock.cc:648] HybridClock initialized: now 1790651770310437 us; error 34 us; skew 500 ppm
I20260929 03:16:10.312417 23490 webserver.cc:460] Webserver started at http://127.13.238.190:37703/ using document root <none> and password file <none>
I20260929 03:16:10.313090 23490 fs_manager.cc:359] Metadata directory not provided
I20260929 03:16:10.313158 23490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:16:10.313393 23490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:16:10.315933 23490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "b7a309e533474fd299ac80d1f1e450b9"
format_stamp: "Formatted at 2026-09-29 03:16:10 on dist-test-slave-nbjg"
server_key: "d3187e89b3f50066544e7bbf96f08b23"
server_key_iv: "57bb154a0342d0b7299fa123496ce2fa"
server_key_version: "encryptionkey@0"
I20260929 03:16:10.316553 23490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "b7a309e533474fd299ac80d1f1e450b9"
format_stamp: "Formatted at 2026-09-29 03:16:10 on dist-test-slave-nbjg"
server_key: "d3187e89b3f50066544e7bbf96f08b23"
server_key_iv: "57bb154a0342d0b7299fa123496ce2fa"
server_key_version: "encryptionkey@0"
I20260929 03:16:10.320345 23490 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:16:10.322866 23504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:10.324282 23490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 03:16:10.324411 23490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "b7a309e533474fd299ac80d1f1e450b9"
format_stamp: "Formatted at 2026-09-29 03:16:10 on dist-test-slave-nbjg"
server_key: "d3187e89b3f50066544e7bbf96f08b23"
server_key_iv: "57bb154a0342d0b7299fa123496ce2fa"
server_key_version: "encryptionkey@0"
I20260929 03:16:10.324533 23490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:16:10.357154 23490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:16:10.357932 23490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:16:10.358111 23490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:16:10.364849 23490 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:35893
I20260929 03:16:10.364866 23556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:35893 every 8 connection(s)
I20260929 03:16:10.365849 23490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:16:10.369028 23557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:16:10.371809 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 23490
I20260929 03:16:10.371913 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:16:10.372172 14266 external_mini_cluster.cc:1428] Setting key f93254a399df2a4c7e645195bcdaa109
I20260929 03:16:10.375036 23557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7a309e533474fd299ac80d1f1e450b9: Bootstrap starting.
I20260929 03:16:10.377549 23557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7a309e533474fd299ac80d1f1e450b9: Neither blocks nor log segments found. Creating new log.
I20260929 03:16:10.378463 23557 log.cc:824] T 00000000000000000000000000000000 P b7a309e533474fd299ac80d1f1e450b9: Log is configured to *not* fsync() on all Append() calls
I20260929 03:16:10.380715 23557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7a309e533474fd299ac80d1f1e450b9: No bootstrap required, opened a new log
I20260929 03:16:10.381723 23560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:10.373612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33614 (local address 127.13.238.190:35893)
0929 03:16:10.374271 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:10.374287 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:10.374326 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:16:10.374945 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:10.374962 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:10.375412 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:10.375698 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:10.376046 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:10.377009 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:10.378243 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:10.378740 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:10.378891 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:10.379683 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:16:10.379689 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:16:10.379706 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:16:10.380190 (+ 484us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:16:10.380260 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:16:10.380287 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:16:10.380387 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 03:16:10.380485 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 03:16:10.380774 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":117,"threads_started":1}
I20260929 03:16:10.384450 23557 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7a309e533474fd299ac80d1f1e450b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a309e533474fd299ac80d1f1e450b9" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 35893 } }
I20260929 03:16:10.384666 23557 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7a309e533474fd299ac80d1f1e450b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:16:10.384706 23557 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7a309e533474fd299ac80d1f1e450b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7a309e533474fd299ac80d1f1e450b9, State: Initialized, Role: FOLLOWER
I20260929 03:16:10.385082 23557 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7a309e533474fd299ac80d1f1e450b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a309e533474fd299ac80d1f1e450b9" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 35893 } }
I20260929 03:16:10.385215 23557 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7a309e533474fd299ac80d1f1e450b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:16:10.385262 23557 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7a309e533474fd299ac80d1f1e450b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:16:10.385332 23557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7a309e533474fd299ac80d1f1e450b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:16:10.386184 23557 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7a309e533474fd299ac80d1f1e450b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a309e533474fd299ac80d1f1e450b9" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 35893 } }
I20260929 03:16:10.386503 23557 leader_election.cc:302] T 00000000000000000000000000000000 P b7a309e533474fd299ac80d1f1e450b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7a309e533474fd299ac80d1f1e450b9; no voters:
I20260929 03:16:10.386750 23557 leader_election.cc:288] T 00000000000000000000000000000000 P b7a309e533474fd299ac80d1f1e450b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:16:10.386845 23562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7a309e533474fd299ac80d1f1e450b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:16:10.387063 23562 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7a309e533474fd299ac80d1f1e450b9 [term 1 LEADER]: Becoming Leader. State: Replica: b7a309e533474fd299ac80d1f1e450b9, State: Running, Role: LEADER
I20260929 03:16:10.387432 23562 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7a309e533474fd299ac80d1f1e450b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a309e533474fd299ac80d1f1e450b9" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 35893 } }
I20260929 03:16:10.387863 23557 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7a309e533474fd299ac80d1f1e450b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:16:10.389041 23563 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7a309e533474fd299ac80d1f1e450b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7a309e533474fd299ac80d1f1e450b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a309e533474fd299ac80d1f1e450b9" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 35893 } } }
I20260929 03:16:10.389142 23563 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7a309e533474fd299ac80d1f1e450b9 [sys.catalog]: This master's current role is: LEADER
I20260929 03:16:10.389447 23564 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7a309e533474fd299ac80d1f1e450b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7a309e533474fd299ac80d1f1e450b9. Latest consensus state: current_term: 1 leader_uuid: "b7a309e533474fd299ac80d1f1e450b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a309e533474fd299ac80d1f1e450b9" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 35893 } } }
I20260929 03:16:10.389532 23564 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7a309e533474fd299ac80d1f1e450b9 [sys.catalog]: This master's current role is: LEADER
I20260929 03:16:10.389621 23571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:16:10.392226 23571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:16:10.397471 23571 catalog_manager.cc:1347] Generated new cluster ID: da60458230e64d0299110d07db81c237
I20260929 03:16:10.397542 23571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:16:10.406049 23571 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:16:10.407226 23571 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:16:10.415091 23571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7a309e533474fd299ac80d1f1e450b9: Generated new TSK 0
I20260929 03:16:10.415853 23571 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:16:10.426668 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:0
--local_ip_for_outbound_sockets=127.13.238.129
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:35893
--builtin_ntp_servers=127.13.238.148:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 03:16:10.536966 23581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:16:10.537233 23581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:16:10.537282 23581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:16:10.541247 23581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:16:10.541299 23581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:16:10.541371 23581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:16:10.546104 23581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:35893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:16:10.547236 23581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:16:10.548650 23581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:16:10.558272 23589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:16:10.558276 23587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:16:10.558408 23586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:16:10.558756 23581 server_base.cc:1034] running on GCE node
I20260929 03:16:10.559111 23581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:16:10.559721 23581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:16:10.560899 23581 hybrid_clock.cc:648] HybridClock initialized: now 1790651770560865 us; error 42 us; skew 500 ppm
I20260929 03:16:10.562839 23581 webserver.cc:460] Webserver started at http://127.13.238.129:41207/ using document root <none> and password file <none>
I20260929 03:16:10.563392 23581 fs_manager.cc:359] Metadata directory not provided
I20260929 03:16:10.563459 23581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:16:10.563671 23581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:16:10.565701 23581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/data/instance:
uuid: "bc6749c5d5554552a94fabf1bdd44938"
format_stamp: "Formatted at 2026-09-29 03:16:10 on dist-test-slave-nbjg"
server_key: "fcc4e24ed9bf6c12a6769ed3e23cb78d"
server_key_iv: "cdd788e9df4482d635a3725506814c56"
server_key_version: "encryptionkey@0"
I20260929 03:16:10.566262 23581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance:
uuid: "bc6749c5d5554552a94fabf1bdd44938"
format_stamp: "Formatted at 2026-09-29 03:16:10 on dist-test-slave-nbjg"
server_key: "fcc4e24ed9bf6c12a6769ed3e23cb78d"
server_key_iv: "cdd788e9df4482d635a3725506814c56"
server_key_version: "encryptionkey@0"
I20260929 03:16:10.569881 23581 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:16:10.572539 23595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:10.573765 23581 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:16:10.573873 23581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "bc6749c5d5554552a94fabf1bdd44938"
format_stamp: "Formatted at 2026-09-29 03:16:10 on dist-test-slave-nbjg"
server_key: "fcc4e24ed9bf6c12a6769ed3e23cb78d"
server_key_iv: "cdd788e9df4482d635a3725506814c56"
server_key_version: "encryptionkey@0"
I20260929 03:16:10.573975 23581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:16:10.595197 23581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:16:10.595968 23581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:16:10.596148 23581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:16:10.596769 23581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:16:10.597766 23581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:16:10.597813 23581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:10.597867 23581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:16:10.597896 23581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:10.606252 23581 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:34285
I20260929 03:16:10.606279 23708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:34285 every 8 connection(s)
I20260929 03:16:10.607174 23581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:16:10.613001 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 23581
I20260929 03:16:10.613087 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance
I20260929 03:16:10.613327 14266 external_mini_cluster.cc:1428] Setting key d6eec864f39546388c5cb4f9c8169da7
I20260929 03:16:10.615198 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:0
--local_ip_for_outbound_sockets=127.13.238.130
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:35893
--builtin_ntp_servers=127.13.238.148:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 03:16:10.616745 23560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:10.609041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:48181 (local address 127.13.238.190:35893)
0929 03:16:10.609187 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:10.609191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:10.610403 (+ 1212us) server_negotiation.cc:407] Connection header received
0929 03:16:10.611880 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:10.611883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:10.611930 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:10.612010 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:10.613011 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:10.613602 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:10.614951 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:10.615167 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:10.615253 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:10.616319 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:16:10.616324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:16:10.616326 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:16:10.616342 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:16:10.616367 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:16:10.616376 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:16:10.616437 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:16:10.616580 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 03:16:10.616633 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:16:10.617831 23711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:10.609474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:35893 (local address 127.13.238.129:48181)
0929 03:16:10.610252 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:10.610306 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:10.611642 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:10.612212 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:10.612220 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:10.612314 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:10.612858 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:10.612872 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:10.613727 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:10.613731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:10.614849 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:10.614858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:10.615308 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:10.615312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:10.615480 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:10.616025 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:16:10.616027 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:10.616068 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:16:10.616135 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:16:10.616448 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:16:10.616456 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:16:10.616462 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:16:10.616582 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 03:16:10.616889 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":120,"threads_started":1}
I20260929 03:16:10.619118 23709 heartbeater.cc:347] Connected to a master server at 127.13.238.190:35893
I20260929 03:16:10.619344 23709 heartbeater.cc:464] Registering TS with master...
I20260929 03:16:10.619838 23709 heartbeater.cc:511] Master 127.13.238.190:35893 requested a full tablet report, sending...
I20260929 03:16:10.621310 23521 ts_manager.cc:194] Registered new tserver with Master: bc6749c5d5554552a94fabf1bdd44938 (127.13.238.129:34285)
I20260929 03:16:10.622465 23521 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.238.129:48181
W20260929 03:16:10.724771 23712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:16:10.725030 23712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:16:10.725081 23712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:16:10.729228 23712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:16:10.729282 23712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:16:10.729355 23712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:16:10.734089 23712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:35893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:16:10.735200 23712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:16:10.736737 23712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:16:10.746600 23720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:16:10.746572 23717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:16:10.746685 23712 server_base.cc:1034] running on GCE node
W20260929 03:16:10.746578 23718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:16:10.747175 23712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:16:10.747792 23712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:16:10.748942 23712 hybrid_clock.cc:648] HybridClock initialized: now 1790651770748912 us; error 45 us; skew 500 ppm
I20260929 03:16:10.750816 23712 webserver.cc:460] Webserver started at http://127.13.238.130:40649/ using document root <none> and password file <none>
I20260929 03:16:10.751386 23712 fs_manager.cc:359] Metadata directory not provided
I20260929 03:16:10.751453 23712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:16:10.751641 23712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:16:10.753674 23712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/data/instance:
uuid: "b21e81b7951841bd80ece1633a3cd86d"
format_stamp: "Formatted at 2026-09-29 03:16:10 on dist-test-slave-nbjg"
server_key: "3a0a48e99dd8d6dec1095b5fc9f547f4"
server_key_iv: "3b30b0dd6cf31740a1cd962755817429"
server_key_version: "encryptionkey@0"
I20260929 03:16:10.754238 23712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance:
uuid: "b21e81b7951841bd80ece1633a3cd86d"
format_stamp: "Formatted at 2026-09-29 03:16:10 on dist-test-slave-nbjg"
server_key: "3a0a48e99dd8d6dec1095b5fc9f547f4"
server_key_iv: "3b30b0dd6cf31740a1cd962755817429"
server_key_version: "encryptionkey@0"
I20260929 03:16:10.757776 23712 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260929 03:16:10.760236 23726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:10.761415 23712 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:16:10.761523 23712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "b21e81b7951841bd80ece1633a3cd86d"
format_stamp: "Formatted at 2026-09-29 03:16:10 on dist-test-slave-nbjg"
server_key: "3a0a48e99dd8d6dec1095b5fc9f547f4"
server_key_iv: "3b30b0dd6cf31740a1cd962755817429"
server_key_version: "encryptionkey@0"
I20260929 03:16:10.761616 23712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:16:10.775291 23712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:16:10.776145 23712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:16:10.776348 23712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:16:10.777073 23712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:16:10.778226 23712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:16:10.778285 23712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:10.778344 23712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:16:10.778383 23712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:10.786907 23712 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:40755
I20260929 03:16:10.786931 23839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:40755 every 8 connection(s)
I20260929 03:16:10.787887 23712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:16:10.791561 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 23712
I20260929 03:16:10.791658 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance
I20260929 03:16:10.791918 14266 external_mini_cluster.cc:1428] Setting key 102062c3b7f2fcf4eb237175e3df6dde
I20260929 03:16:10.793740 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:0
--local_ip_for_outbound_sockets=127.13.238.131
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:35893
--builtin_ntp_servers=127.13.238.148:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 03:16:10.797314 23560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:10.789753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:59951 (local address 127.13.238.190:35893)
0929 03:16:10.789900 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:10.789904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:10.791062 (+ 1158us) server_negotiation.cc:407] Connection header received
0929 03:16:10.792459 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:10.792463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:10.792519 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:10.792603 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:10.793494 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:10.794057 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:10.795268 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:10.795513 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:10.795606 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:10.796778 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:16:10.796783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:16:10.796786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:16:10.796802 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:16:10.796828 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:16:10.796840 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:16:10.796915 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:16:10.797077 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 03:16:10.797165 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:16:10.798477 23842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:10.790240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:35893 (local address 127.13.238.130:59951)
0929 03:16:10.790918 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:10.790959 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:10.792248 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:10.792776 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:10.792783 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:10.792869 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:10.793360 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:10.793372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:10.794179 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:10.794182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:10.795144 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:10.795156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:10.795654 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:10.795658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:10.795851 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:10.796494 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:16:10.796496 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:10.796547 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:16:10.796626 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:16:10.796930 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:16:10.796938 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:16:10.796943 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:16:10.797056 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 03:16:10.797386 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":113,"threads_started":1}
I20260929 03:16:10.799611 23840 heartbeater.cc:347] Connected to a master server at 127.13.238.190:35893
I20260929 03:16:10.799918 23840 heartbeater.cc:464] Registering TS with master...
I20260929 03:16:10.800601 23840 heartbeater.cc:511] Master 127.13.238.190:35893 requested a full tablet report, sending...
I20260929 03:16:10.801746 23521 ts_manager.cc:194] Registered new tserver with Master: b21e81b7951841bd80ece1633a3cd86d (127.13.238.130:40755)
I20260929 03:16:10.802372 23521 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.238.130:59951
W20260929 03:16:10.905483 23843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:16:10.905740 23843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:16:10.905776 23843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:16:10.909777 23843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:16:10.909834 23843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:16:10.909886 23843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:16:10.914659 23843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:35893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:16:10.915797 23843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:16:10.917125 23843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:16:10.926539 23848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:16:10.926589 23843 server_base.cc:1034] running on GCE node
W20260929 03:16:10.926497 23851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:16:10.926513 23849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:16:10.927078 23843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:16:10.927708 23843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:16:10.928861 23843 hybrid_clock.cc:648] HybridClock initialized: now 1790651770928841 us; error 29 us; skew 500 ppm
I20260929 03:16:10.930688 23843 webserver.cc:460] Webserver started at http://127.13.238.131:38229/ using document root <none> and password file <none>
I20260929 03:16:10.931257 23843 fs_manager.cc:359] Metadata directory not provided
I20260929 03:16:10.931315 23843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:16:10.931516 23843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:16:10.933555 23843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/data/instance:
uuid: "0a86c678352a43418c5ade0012c3e25e"
format_stamp: "Formatted at 2026-09-29 03:16:10 on dist-test-slave-nbjg"
server_key: "7c0198ae090809510f0adfbc43c520d0"
server_key_iv: "b1edfc9e62eb780bd8b9f56010540230"
server_key_version: "encryptionkey@0"
I20260929 03:16:10.934077 23843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance:
uuid: "0a86c678352a43418c5ade0012c3e25e"
format_stamp: "Formatted at 2026-09-29 03:16:10 on dist-test-slave-nbjg"
server_key: "7c0198ae090809510f0adfbc43c520d0"
server_key_iv: "b1edfc9e62eb780bd8b9f56010540230"
server_key_version: "encryptionkey@0"
I20260929 03:16:10.937645 23843 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260929 03:16:10.940124 23857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:10.941352 23843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:16:10.941473 23843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "0a86c678352a43418c5ade0012c3e25e"
format_stamp: "Formatted at 2026-09-29 03:16:10 on dist-test-slave-nbjg"
server_key: "7c0198ae090809510f0adfbc43c520d0"
server_key_iv: "b1edfc9e62eb780bd8b9f56010540230"
server_key_version: "encryptionkey@0"
I20260929 03:16:10.941577 23843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:16:10.967049 23843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:16:10.967780 23843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:16:10.967952 23843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:16:10.968551 23843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:16:10.969534 23843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:16:10.969581 23843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:10.969630 23843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:16:10.969659 23843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:10.978060 23843 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:45427
I20260929 03:16:10.978093 23970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:45427 every 8 connection(s)
I20260929 03:16:10.979063 23843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:16:10.980434 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 23843
I20260929 03:16:10.980525 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance
I20260929 03:16:10.980841 14266 external_mini_cluster.cc:1428] Setting key 562bb2842322237b2520f59669ef0afa
I20260929 03:16:10.988461 23560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:10.980962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:51583 (local address 127.13.238.190:35893)
0929 03:16:10.981126 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:10.981131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:10.982421 (+ 1290us) server_negotiation.cc:407] Connection header received
0929 03:16:10.983906 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:10.983908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:10.983954 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:10.984016 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:10.984887 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:10.985424 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:10.986612 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:10.986840 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:10.986927 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:10.987926 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:16:10.987950 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:16:10.987954 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:16:10.987969 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:16:10.987993 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:16:10.988004 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:16:10.988069 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:16:10.988269 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 03:16:10.988328 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 03:16:10.989372 23973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:10.981402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:35893 (local address 127.13.238.131:51583)
0929 03:16:10.982268 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:10.982328 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:10.983711 (+ 1383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:10.984191 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:10.984198 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:10.984286 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:10.984783 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:10.984796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:10.985513 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:10.985515 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:10.986489 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:10.986499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:10.986967 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:10.986970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:10.987114 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:10.987667 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:16:10.987669 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:10.987711 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:16:10.987777 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:16:10.988084 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:16:10.988092 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:16:10.988098 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:16:10.988221 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 03:16:10.988457 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":141,"threads_started":1}
I20260929 03:16:10.990641 23971 heartbeater.cc:347] Connected to a master server at 127.13.238.190:35893
I20260929 03:16:10.990922 23971 heartbeater.cc:464] Registering TS with master...
I20260929 03:16:10.991477 23971 heartbeater.cc:511] Master 127.13.238.190:35893 requested a full tablet report, sending...
I20260929 03:16:10.992491 23521 ts_manager.cc:194] Registered new tserver with Master: 0a86c678352a43418c5ade0012c3e25e (127.13.238.131:45427)
I20260929 03:16:10.993075 23521 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.238.131:51583
I20260929 03:16:10.994630 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 03:16:10.997210 23560 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:10.996394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33626 (local address 127.13.238.190:35893)
0929 03:16:10.996500 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:10.996504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:10.996685 (+ 181us) server_negotiation.cc:407] Connection header received
0929 03:16:10.996779 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:10.996781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:10.996822 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:10.996885 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:10.997068 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33626: BlockingRecv error: recv got EOF from 127.0.0.1:33626 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:16:10.997629 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 23581
I20260929 03:16:11.002938 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 23712
I20260929 03:16:11.007701 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 23843
I20260929 03:16:11.012280 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 23490
2026-09-29T03:16:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (863 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T03:16:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:16:11Z Disabled control of system clock
I20260929 03:16:11.040020 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:34255
--webserver_interface=127.13.238.190
--webserver_port=0
--builtin_ntp_servers=127.13.238.148:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:34255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 03:16:11.149037 23982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:16:11.149314 23982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:16:11.149353 23982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:16:11.153409 23982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:16:11.153475 23982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:16:11.153489 23982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:16:11.153508 23982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:16:11.153537 23982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:16:11.158423 23982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:34255
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:34255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:16:11.159513 23982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:16:11.160982 23982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:16:11.169301 23982 server_base.cc:1034] running on GCE node
W20260929 03:16:11.169286 23987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:16:11.169265 23990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:16:11.169265 23988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:16:11.170009 23982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:16:11.171133 23982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:16:11.172333 23982 hybrid_clock.cc:648] HybridClock initialized: now 1790651771172309 us; error 33 us; skew 500 ppm
I20260929 03:16:11.174261 23982 webserver.cc:460] Webserver started at http://127.13.238.190:34299/ using document root <none> and password file <none>
I20260929 03:16:11.174829 23982 fs_manager.cc:359] Metadata directory not provided
I20260929 03:16:11.174898 23982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:16:11.175100 23982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:16:11.177134 23982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "074ede06945e489886ea2857b799c6b3"
format_stamp: "Formatted at 2026-09-29 03:16:11 on dist-test-slave-nbjg"
server_key: "614db4d9b58022f94e8191c49cc535de"
server_key_iv: "b372ae65e83ae546a1ce23c5ee40bbaf"
server_key_version: "encryptionkey@0"
I20260929 03:16:11.177698 23982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "074ede06945e489886ea2857b799c6b3"
format_stamp: "Formatted at 2026-09-29 03:16:11 on dist-test-slave-nbjg"
server_key: "614db4d9b58022f94e8191c49cc535de"
server_key_iv: "b372ae65e83ae546a1ce23c5ee40bbaf"
server_key_version: "encryptionkey@0"
I20260929 03:16:11.181443 23982 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:16:11.183912 23996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:11.185148 23982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 03:16:11.185266 23982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "074ede06945e489886ea2857b799c6b3"
format_stamp: "Formatted at 2026-09-29 03:16:11 on dist-test-slave-nbjg"
server_key: "614db4d9b58022f94e8191c49cc535de"
server_key_iv: "b372ae65e83ae546a1ce23c5ee40bbaf"
server_key_version: "encryptionkey@0"
I20260929 03:16:11.185360 23982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:16:11.189860 23982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:16:11.190492 23982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:16:11.190646 23982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:16:11.197026 23982 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:34255
I20260929 03:16:11.197031 24048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:34255 every 8 connection(s)
I20260929 03:16:11.197955 23982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:16:11.201058 24049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:16:11.205005 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 23982
I20260929 03:16:11.205114 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:16:11.205384 14266 external_mini_cluster.cc:1428] Setting key 4b679ef39faa08d364abbbeeb6ef1ff4
I20260929 03:16:11.208019 24049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 074ede06945e489886ea2857b799c6b3: Bootstrap starting.
I20260929 03:16:11.210546 24049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 074ede06945e489886ea2857b799c6b3: Neither blocks nor log segments found. Creating new log.
I20260929 03:16:11.211316 24049 log.cc:824] T 00000000000000000000000000000000 P 074ede06945e489886ea2857b799c6b3: Log is configured to *not* fsync() on all Append() calls
I20260929 03:16:11.212057 24052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:11.206848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33486 (local address 127.13.238.190:34255)
0929 03:16:11.207682 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:11.207704 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:11.207750 (+ 46us) server_negotiation.cc:407] Connection header received
0929 03:16:11.208627 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:11.208653 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:11.209130 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:11.209479 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:11.210722 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:16:11.210728 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:16:11.210739 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:16:11.210751 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:16:11.210805 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:16:11.210821 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:16:11.210916 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 03:16:11.211028 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 03:16:11.211265 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":124,"threads_started":1}
I20260929 03:16:11.213297 24049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 074ede06945e489886ea2857b799c6b3: No bootstrap required, opened a new log
I20260929 03:16:11.216814 24049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 074ede06945e489886ea2857b799c6b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074ede06945e489886ea2857b799c6b3" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 34255 } }
I20260929 03:16:11.217031 24049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 074ede06945e489886ea2857b799c6b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:16:11.217085 24049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 074ede06945e489886ea2857b799c6b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074ede06945e489886ea2857b799c6b3, State: Initialized, Role: FOLLOWER
I20260929 03:16:11.217470 24049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 074ede06945e489886ea2857b799c6b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074ede06945e489886ea2857b799c6b3" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 34255 } }
I20260929 03:16:11.217609 24049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 074ede06945e489886ea2857b799c6b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:16:11.217672 24049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 074ede06945e489886ea2857b799c6b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:16:11.217775 24049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 074ede06945e489886ea2857b799c6b3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:16:11.218670 24049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 074ede06945e489886ea2857b799c6b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074ede06945e489886ea2857b799c6b3" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 34255 } }
I20260929 03:16:11.218991 24049 leader_election.cc:302] T 00000000000000000000000000000000 P 074ede06945e489886ea2857b799c6b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 074ede06945e489886ea2857b799c6b3; no voters:
I20260929 03:16:11.219266 24049 leader_election.cc:288] T 00000000000000000000000000000000 P 074ede06945e489886ea2857b799c6b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:16:11.219352 24054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 074ede06945e489886ea2857b799c6b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:16:11.219542 24054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 074ede06945e489886ea2857b799c6b3 [term 1 LEADER]: Becoming Leader. State: Replica: 074ede06945e489886ea2857b799c6b3, State: Running, Role: LEADER
I20260929 03:16:11.219857 24054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 074ede06945e489886ea2857b799c6b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074ede06945e489886ea2857b799c6b3" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 34255 } }
I20260929 03:16:11.220449 24049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 074ede06945e489886ea2857b799c6b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:16:11.221401 24055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 074ede06945e489886ea2857b799c6b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "074ede06945e489886ea2857b799c6b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074ede06945e489886ea2857b799c6b3" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 34255 } } }
I20260929 03:16:11.221529 24055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 074ede06945e489886ea2857b799c6b3 [sys.catalog]: This master's current role is: LEADER
I20260929 03:16:11.221800 24056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 074ede06945e489886ea2857b799c6b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 074ede06945e489886ea2857b799c6b3. Latest consensus state: current_term: 1 leader_uuid: "074ede06945e489886ea2857b799c6b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074ede06945e489886ea2857b799c6b3" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 34255 } } }
I20260929 03:16:11.221871 24056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 074ede06945e489886ea2857b799c6b3 [sys.catalog]: This master's current role is: LEADER
I20260929 03:16:11.222110 24063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:16:11.225350 24063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:16:11.231305 24063 catalog_manager.cc:1347] Generated new cluster ID: a0da6da684754a1995e4206e2d9fbba2
I20260929 03:16:11.231372 24063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:16:11.256228 24063 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:16:11.256572 24063 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:16:11.263543 24063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 074ede06945e489886ea2857b799c6b3: Generated new TSK 0
I20260929 03:16:11.264246 24063 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:16:11.270154 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:0
--local_ip_for_outbound_sockets=127.13.238.129
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:34255
--builtin_ntp_servers=127.13.238.148:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 03:16:11.382395 24073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:16:11.382668 24073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:16:11.382707 24073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:16:11.386766 24073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:16:11.386816 24073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:16:11.386873 24073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:16:11.391659 24073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:34255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:16:11.392705 24073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:16:11.394086 24073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:16:11.403872 24079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:16:11.403885 24081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:16:11.404026 24073 server_base.cc:1034] running on GCE node
W20260929 03:16:11.403936 24078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:16:11.404547 24073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:16:11.405196 24073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:16:11.406358 24073 hybrid_clock.cc:648] HybridClock initialized: now 1790651771406333 us; error 34 us; skew 500 ppm
I20260929 03:16:11.412292 24073 webserver.cc:460] Webserver started at http://127.13.238.129:36827/ using document root <none> and password file <none>
I20260929 03:16:11.412896 24073 fs_manager.cc:359] Metadata directory not provided
I20260929 03:16:11.412981 24073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:16:11.413215 24073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:16:11.415350 24073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/data/instance:
uuid: "31899ea68fb241d9a6dcbb34d646e524"
format_stamp: "Formatted at 2026-09-29 03:16:11 on dist-test-slave-nbjg"
server_key: "4f49da6453df7c660db43b399665d533"
server_key_iv: "864659ef34262c768df2e1045f7a5cac"
server_key_version: "encryptionkey@0"
I20260929 03:16:11.415872 24073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance:
uuid: "31899ea68fb241d9a6dcbb34d646e524"
format_stamp: "Formatted at 2026-09-29 03:16:11 on dist-test-slave-nbjg"
server_key: "4f49da6453df7c660db43b399665d533"
server_key_iv: "864659ef34262c768df2e1045f7a5cac"
server_key_version: "encryptionkey@0"
I20260929 03:16:11.419505 24073 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:16:11.422019 24087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:11.423259 24073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:16:11.423367 24073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "31899ea68fb241d9a6dcbb34d646e524"
format_stamp: "Formatted at 2026-09-29 03:16:11 on dist-test-slave-nbjg"
server_key: "4f49da6453df7c660db43b399665d533"
server_key_iv: "864659ef34262c768df2e1045f7a5cac"
server_key_version: "encryptionkey@0"
I20260929 03:16:11.423465 24073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:16:11.427851 24073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:16:11.428432 24073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:16:11.428578 24073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:16:11.429080 24073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:16:11.429953 24073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:16:11.430004 24073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:11.430053 24073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:16:11.430078 24073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:11.438076 24073 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:38367
I20260929 03:16:11.438099 24200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:38367 every 8 connection(s)
I20260929 03:16:11.438963 24073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:16:11.444854 24052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:11.440670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:48965 (local address 127.13.238.190:34255)
0929 03:16:11.440800 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:11.440803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:11.441864 (+ 1061us) server_negotiation.cc:407] Connection header received
0929 03:16:11.443240 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:11.443244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:11.443289 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:11.443359 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:11.444412 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:16:11.444416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:16:11.444419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:16:11.444432 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:16:11.444454 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:16:11.444462 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:16:11.444540 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:16:11.444669 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 03:16:11.444716 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 03:16:11.445720 24203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:11.441071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:34255 (local address 127.13.238.129:48965)
0929 03:16:11.441723 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:11.441762 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:11.443011 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:11.443522 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:11.443529 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:11.443625 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:11.444142 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:16:11.444144 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:11.444192 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:16:11.444265 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:16:11.444530 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:16:11.444538 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:16:11.444544 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:16:11.444700 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 03:16:11.444944 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":111,"threads_started":1}
I20260929 03:16:11.446056 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 24073
I20260929 03:16:11.446164 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance
I20260929 03:16:11.446460 14266 external_mini_cluster.cc:1428] Setting key 6563f04e79f5564c279e1113bc4fff19
I20260929 03:16:11.446940 24201 heartbeater.cc:347] Connected to a master server at 127.13.238.190:34255
I20260929 03:16:11.447147 24201 heartbeater.cc:464] Registering TS with master...
I20260929 03:16:11.447521 24201 heartbeater.cc:511] Master 127.13.238.190:34255 requested a full tablet report, sending...
I20260929 03:16:11.448335 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:0
--local_ip_for_outbound_sockets=127.13.238.130
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:34255
--builtin_ntp_servers=127.13.238.148:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 03:16:11.449102 24013 ts_manager.cc:194] Registered new tserver with Master: 31899ea68fb241d9a6dcbb34d646e524 (127.13.238.129:38367)
W20260929 03:16:11.559116 24204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:16:11.559408 24204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:16:11.559465 24204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:16:11.563527 24204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:16:11.563582 24204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:16:11.563660 24204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:16:11.568391 24204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:34255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:16:11.569442 24204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:16:11.570875 24204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:16:11.580399 24209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:16:11.580462 24204 server_base.cc:1034] running on GCE node
W20260929 03:16:11.580400 24212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:16:11.580400 24210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:16:11.581048 24204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:16:11.581724 24204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:16:11.582894 24204 hybrid_clock.cc:648] HybridClock initialized: now 1790651771582872 us; error 31 us; skew 500 ppm
I20260929 03:16:11.584818 24204 webserver.cc:460] Webserver started at http://127.13.238.130:39775/ using document root <none> and password file <none>
I20260929 03:16:11.585400 24204 fs_manager.cc:359] Metadata directory not provided
I20260929 03:16:11.585467 24204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:16:11.585665 24204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:16:11.587801 24204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/data/instance:
uuid: "3da921a29f1046a9a84c0fa1be097ef2"
format_stamp: "Formatted at 2026-09-29 03:16:11 on dist-test-slave-nbjg"
server_key: "739cca2b77c418ebb941dab3eb1ab91b"
server_key_iv: "87e6e44e0f2973510342185e57461248"
server_key_version: "encryptionkey@0"
I20260929 03:16:11.588361 24204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance:
uuid: "3da921a29f1046a9a84c0fa1be097ef2"
format_stamp: "Formatted at 2026-09-29 03:16:11 on dist-test-slave-nbjg"
server_key: "739cca2b77c418ebb941dab3eb1ab91b"
server_key_iv: "87e6e44e0f2973510342185e57461248"
server_key_version: "encryptionkey@0"
I20260929 03:16:11.592257 24204 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 03:16:11.594799 24218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:11.596043 24204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:16:11.596151 24204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "3da921a29f1046a9a84c0fa1be097ef2"
format_stamp: "Formatted at 2026-09-29 03:16:11 on dist-test-slave-nbjg"
server_key: "739cca2b77c418ebb941dab3eb1ab91b"
server_key_iv: "87e6e44e0f2973510342185e57461248"
server_key_version: "encryptionkey@0"
I20260929 03:16:11.596256 24204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:16:11.600751 24204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:16:11.601336 24204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:16:11.601487 24204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:16:11.602003 24204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:16:11.602926 24204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:16:11.602972 24204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:11.603021 24204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:16:11.603048 24204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:11.611111 24204 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:38381
I20260929 03:16:11.611131 24331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:38381 every 8 connection(s)
I20260929 03:16:11.611989 24204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:16:11.614130 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 24204
I20260929 03:16:11.614261 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance
I20260929 03:16:11.614604 14266 external_mini_cluster.cc:1428] Setting key 59b6e0015dee32c1936bf099c1309331
I20260929 03:16:11.616591 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:0
--local_ip_for_outbound_sockets=127.13.238.131
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:34255
--builtin_ntp_servers=127.13.238.148:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 03:16:11.618551 24052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:11.613943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:47377 (local address 127.13.238.190:34255)
0929 03:16:11.614071 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:11.614074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:11.615419 (+ 1345us) server_negotiation.cc:407] Connection header received
0929 03:16:11.616845 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:11.616849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:11.616893 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:11.616967 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:11.618086 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:16:11.618091 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:16:11.618093 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:16:11.618109 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:16:11.618134 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:16:11.618145 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:16:11.618259 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 03:16:11.618365 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 03:16:11.618413 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 03:16:11.619377 24334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:11.614488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:34255 (local address 127.13.238.130:47377)
0929 03:16:11.615277 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:11.615320 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:11.616578 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:11.617127 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:11.617135 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:11.617237 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:11.617820 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:16:11.617821 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:11.617862 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:16:11.617934 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:16:11.618249 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:16:11.618254 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:16:11.618259 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:16:11.618388 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 03:16:11.618586 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":138,"threads_started":1}
I20260929 03:16:11.620492 24332 heartbeater.cc:347] Connected to a master server at 127.13.238.190:34255
I20260929 03:16:11.620702 24332 heartbeater.cc:464] Registering TS with master...
I20260929 03:16:11.621110 24332 heartbeater.cc:511] Master 127.13.238.190:34255 requested a full tablet report, sending...
I20260929 03:16:11.622273 24013 ts_manager.cc:194] Registered new tserver with Master: 3da921a29f1046a9a84c0fa1be097ef2 (127.13.238.130:38381)
W20260929 03:16:11.726545 24335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:16:11.726819 24335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:16:11.726866 24335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:16:11.730986 24335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:16:11.731040 24335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:16:11.731105 24335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:16:11.736191 24335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:34255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:16:11.737282 24335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:16:11.738761 24335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:16:11.748008 24343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:16:11.748008 24341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:16:11.748018 24340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:16:11.748266 24335 server_base.cc:1034] running on GCE node
I20260929 03:16:11.748759 24335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:16:11.749370 24335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:16:11.750514 24335 hybrid_clock.cc:648] HybridClock initialized: now 1790651771750487 us; error 38 us; skew 500 ppm
I20260929 03:16:11.752288 24335 webserver.cc:460] Webserver started at http://127.13.238.131:43759/ using document root <none> and password file <none>
I20260929 03:16:11.752840 24335 fs_manager.cc:359] Metadata directory not provided
I20260929 03:16:11.752898 24335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:16:11.753079 24335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:16:11.755148 24335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/data/instance:
uuid: "ded682748f314a479390f7153ee4aa3f"
format_stamp: "Formatted at 2026-09-29 03:16:11 on dist-test-slave-nbjg"
server_key: "eb03f40f9c25ec7d2d78f337b7ec51a6"
server_key_iv: "07171c76d97da49f2bca65a73f18b30a"
server_key_version: "encryptionkey@0"
I20260929 03:16:11.755669 24335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance:
uuid: "ded682748f314a479390f7153ee4aa3f"
format_stamp: "Formatted at 2026-09-29 03:16:11 on dist-test-slave-nbjg"
server_key: "eb03f40f9c25ec7d2d78f337b7ec51a6"
server_key_iv: "07171c76d97da49f2bca65a73f18b30a"
server_key_version: "encryptionkey@0"
I20260929 03:16:11.759297 24335 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 03:16:11.762007 24349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:11.763233 24335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:16:11.763356 24335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "ded682748f314a479390f7153ee4aa3f"
format_stamp: "Formatted at 2026-09-29 03:16:11 on dist-test-slave-nbjg"
server_key: "eb03f40f9c25ec7d2d78f337b7ec51a6"
server_key_iv: "07171c76d97da49f2bca65a73f18b30a"
server_key_version: "encryptionkey@0"
I20260929 03:16:11.763463 24335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:16:11.768421 24335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:16:11.769052 24335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:16:11.769208 24335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:16:11.769748 24335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:16:11.770712 24335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:16:11.770758 24335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:11.770808 24335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:16:11.770833 24335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:11.779114 24335 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:40179
I20260929 03:16:11.779145 24462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:40179 every 8 connection(s)
I20260929 03:16:11.779991 24335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:16:11.782366 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 24335
I20260929 03:16:11.782444 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance
I20260929 03:16:11.782683 14266 external_mini_cluster.cc:1428] Setting key c129de25b60fc6570752d91d9dc67b8c
I20260929 03:16:11.785982 24052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:11.781816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:54407 (local address 127.13.238.190:34255)
0929 03:16:11.781955 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:11.781959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:11.783132 (+ 1173us) server_negotiation.cc:407] Connection header received
0929 03:16:11.784416 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:11.784419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:11.784462 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:11.784545 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:11.785563 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:16:11.785567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:16:11.785569 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:16:11.785581 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:16:11.785600 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:16:11.785607 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:16:11.785680 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:16:11.785808 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 03:16:11.785852 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:16:11.786862 24465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:11.782262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:34255 (local address 127.13.238.131:54407)
0929 03:16:11.782972 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:11.783036 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:11.784221 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:11.784703 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:11.784710 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:11.784798 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:11.785264 (+ 466us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:16:11.785266 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:11.785307 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:16:11.785375 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:16:11.785680 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:16:11.785689 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:16:11.785696 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:16:11.785847 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 03:16:11.786104 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":123,"threads_started":1}
I20260929 03:16:11.787865 24463 heartbeater.cc:347] Connected to a master server at 127.13.238.190:34255
I20260929 03:16:11.788085 24463 heartbeater.cc:464] Registering TS with master...
I20260929 03:16:11.788466 24463 heartbeater.cc:511] Master 127.13.238.190:34255 requested a full tablet report, sending...
I20260929 03:16:11.789405 24013 ts_manager.cc:194] Registered new tserver with Master: ded682748f314a479390f7153ee4aa3f (127.13.238.131:40179)
I20260929 03:16:11.796197 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 03:16:11.798910 24052 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:16:11.798015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33502 (local address 127.13.238.190:34255)
0929 03:16:11.798163 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:11.798167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:11.798350 (+ 183us) server_negotiation.cc:407] Connection header received
0929 03:16:11.798446 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:11.798448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:11.798482 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:11.798585 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:11.798756 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33502: BlockingRecv error: recv got EOF from 127.0.0.1:33502 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 03:16:11.798995 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 24073
I20260929 03:16:11.804409 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 24204
I20260929 03:16:11.809134 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 24335
I20260929 03:16:11.815270 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 23982
2026-09-29T03:16:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (802 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:16:11 dist-test-slave-nbjg krb5kdc[24473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:16:11 dist-test-slave-nbjg krb5kdc[24473](info): set up 2 sockets
Sep 29 03:16:11 dist-test-slave-nbjg krb5kdc[24473](info): commencing operation
krb5kdc: starting...
W20260929 03:16:12.767184 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:16:12 dist-test-slave-nbjg krb5kdc[24473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:16:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:16:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:16:12.885648 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:39579
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:39579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:16:12.995589 24486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:16:12.995896 24486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:16:12.995939 24486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:16:13.000171 24486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:16:13.000253 24486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:16:13.000272 24486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:16:13.000288 24486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:16:13.000301 24486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:16:13.005867 24486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:39579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:39579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:16:13.007051 24486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:16:13.008100 24486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:16:13.016481 24494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:16:13.016496 24491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:16:13.016481 24492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:16:13.016865 24486 server_base.cc:1034] running on GCE node
I20260929 03:16:13.017421 24486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:16:13.018576 24486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:16:13.019742 24486 hybrid_clock.cc:648] HybridClock initialized: now 1790651773019720 us; error 31 us; skew 500 ppm
Sep 29 03:16:13 dist-test-slave-nbjg krb5kdc[24473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651773, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:16:13.022553 24486 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:16:13.023516 24486 webserver.cc:460] Webserver started at http://127.13.238.190:39935/ using document root <none> and password file <none>
I20260929 03:16:13.024107 24486 fs_manager.cc:359] Metadata directory not provided
I20260929 03:16:13.024178 24486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:16:13.024387 24486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:16:13.026754 24486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "257cf5367b2a484da6e660454cbe596d"
format_stamp: "Formatted at 2026-09-29 03:16:13 on dist-test-slave-nbjg"
server_key: "3c191840b7b4a279c83853eae98269db"
server_key_iv: "fe927f84ed5386ca106ee420e3182607"
server_key_version: "encryptionkey@0"
I20260929 03:16:13.027351 24486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "257cf5367b2a484da6e660454cbe596d"
format_stamp: "Formatted at 2026-09-29 03:16:13 on dist-test-slave-nbjg"
server_key: "3c191840b7b4a279c83853eae98269db"
server_key_iv: "fe927f84ed5386ca106ee420e3182607"
server_key_version: "encryptionkey@0"
I20260929 03:16:13.031312 24486 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 03:16:13.033802 24501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:13.035101 24486 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:16:13.035223 24486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "257cf5367b2a484da6e660454cbe596d"
format_stamp: "Formatted at 2026-09-29 03:16:13 on dist-test-slave-nbjg"
server_key: "3c191840b7b4a279c83853eae98269db"
server_key_iv: "fe927f84ed5386ca106ee420e3182607"
server_key_version: "encryptionkey@0"
I20260929 03:16:13.035323 24486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:16:13.052652 24486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:16:13.055572 24486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:16:13.055763 24486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:16:13.062498 24486 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:39579
I20260929 03:16:13.062785 24553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:39579 every 8 connection(s)
I20260929 03:16:13.063465 24486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:16:13.066295 24554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:16:13.071516 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 24486
I20260929 03:16:13.071602 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:16:13.071792 24554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 257cf5367b2a484da6e660454cbe596d: Bootstrap starting.
I20260929 03:16:13.071846 14266 external_mini_cluster.cc:1428] Setting key 1633326a9d9e8853e21279c0c3a843f1
I20260929 03:16:13.074463 24554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 257cf5367b2a484da6e660454cbe596d: Neither blocks nor log segments found. Creating new log.
I20260929 03:16:13.075335 24554 log.cc:824] T 00000000000000000000000000000000 P 257cf5367b2a484da6e660454cbe596d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:16:13.077797 24554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 257cf5367b2a484da6e660454cbe596d: No bootstrap required, opened a new log
Sep 29 03:16:13 dist-test-slave-nbjg krb5kdc[24473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:16:13.081609 24554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 257cf5367b2a484da6e660454cbe596d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257cf5367b2a484da6e660454cbe596d" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 39579 } }
I20260929 03:16:13.081885 24554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 257cf5367b2a484da6e660454cbe596d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:16:13.081966 24554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 257cf5367b2a484da6e660454cbe596d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 257cf5367b2a484da6e660454cbe596d, State: Initialized, Role: FOLLOWER
I20260929 03:16:13.082504 24554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 257cf5367b2a484da6e660454cbe596d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257cf5367b2a484da6e660454cbe596d" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 39579 } }
I20260929 03:16:13.082674 24554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 257cf5367b2a484da6e660454cbe596d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:16:13.082741 24554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 257cf5367b2a484da6e660454cbe596d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:16:13.082845 24554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 257cf5367b2a484da6e660454cbe596d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:16:13.083997 24554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 257cf5367b2a484da6e660454cbe596d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257cf5367b2a484da6e660454cbe596d" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 39579 } }
I20260929 03:16:13.084352 24554 leader_election.cc:302] T 00000000000000000000000000000000 P 257cf5367b2a484da6e660454cbe596d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 257cf5367b2a484da6e660454cbe596d; no voters:
I20260929 03:16:13.084740 24554 leader_election.cc:288] T 00000000000000000000000000000000 P 257cf5367b2a484da6e660454cbe596d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:16:13.084856 24559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 257cf5367b2a484da6e660454cbe596d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:16:13.085064 24559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 257cf5367b2a484da6e660454cbe596d [term 1 LEADER]: Becoming Leader. State: Replica: 257cf5367b2a484da6e660454cbe596d, State: Running, Role: LEADER
I20260929 03:16:13.085393 24559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 257cf5367b2a484da6e660454cbe596d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257cf5367b2a484da6e660454cbe596d" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 39579 } }
I20260929 03:16:13.086338 24554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 257cf5367b2a484da6e660454cbe596d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:16:13.087302 24560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 257cf5367b2a484da6e660454cbe596d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "257cf5367b2a484da6e660454cbe596d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257cf5367b2a484da6e660454cbe596d" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 39579 } } }
I20260929 03:16:13.087428 24560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 257cf5367b2a484da6e660454cbe596d [sys.catalog]: This master's current role is: LEADER
I20260929 03:16:13.087285 24561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 257cf5367b2a484da6e660454cbe596d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 257cf5367b2a484da6e660454cbe596d. Latest consensus state: current_term: 1 leader_uuid: "257cf5367b2a484da6e660454cbe596d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257cf5367b2a484da6e660454cbe596d" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 39579 } } }
I20260929 03:16:13.087529 24561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 257cf5367b2a484da6e660454cbe596d [sys.catalog]: This master's current role is: LEADER
I20260929 03:16:13.087349 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:13.073172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41696 (local address 127.13.238.190:39579)
0929 03:16:13.073612 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:13.073634 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:13.073677 (+ 43us) server_negotiation.cc:407] Connection header received
0929 03:16:13.074535 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:13.074560 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:13.075002 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:13.075389 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:13.076187 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.077746 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.078653 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.079327 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.079493 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:13.081802 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:16:13.081809 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:16:13.081828 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:16:13.081861 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:16:13.083905 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:16:13.084358 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:16:13.084364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:16:13.084371 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:16:13.084514 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:16:13.084815 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:16:13.084818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:16:13.084820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:16:13.085345 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:16:13.085533 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 03:16:13.085810 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 03:16:13.086324 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":110,"threads_started":1}
I20260929 03:16:13.087816 24568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:16:13.091018 24568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:16:13.096606 24568 catalog_manager.cc:1347] Generated new cluster ID: 76f19bb58bb84974acc9df7b17c3e6ad
I20260929 03:16:13.096675 24568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:16:13.103777 24568 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:16:13.104717 24568 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:16:13.112080 24568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 257cf5367b2a484da6e660454cbe596d: Generated new TSK 0
I20260929 03:16:13.112762 24568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
I20260929 03:16:13.151230 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:0
--local_ip_for_outbound_sockets=127.13.238.129
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:39579
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:16:13.271240 24580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:16:13.271515 24580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:16:13.271559 24580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:16:13.275641 24580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:16:13.275713 24580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:16:13.275789 24580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:16:13.281219 24580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:39579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:16:13.282333 24580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:16:13.283377 24580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:16:13.293133 24580 server_base.cc:1034] running on GCE node
W20260929 03:16:13.293079 24586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:16:13.293069 24585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:16:13.293088 24588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:16:13.293769 24580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:16:13.294462 24580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:16:13.295643 24580 hybrid_clock.cc:648] HybridClock initialized: now 1790651773295620 us; error 33 us; skew 500 ppm
Sep 29 03:16:13 dist-test-slave-nbjg krb5kdc[24473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651773, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:16:13.298429 24580 init.cc:377] Logged in from keytab as kudu/127.13.238.129@KRBTEST.COM (short username kudu)
I20260929 03:16:13.299264 24580 webserver.cc:460] Webserver started at http://127.13.238.129:35255/ using document root <none> and password file <none>
I20260929 03:16:13.299813 24580 fs_manager.cc:359] Metadata directory not provided
I20260929 03:16:13.299877 24580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:16:13.300076 24580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:16:13.302101 24580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/data/instance:
uuid: "463753b4f63d4e0c9b555f21518294ff"
format_stamp: "Formatted at 2026-09-29 03:16:13 on dist-test-slave-nbjg"
server_key: "4adde61997e5ca456f86c16f81eda154"
server_key_iv: "ac7d1bd6d45c4d52cf5b9332f10634fa"
server_key_version: "encryptionkey@0"
I20260929 03:16:13.302688 24580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance:
uuid: "463753b4f63d4e0c9b555f21518294ff"
format_stamp: "Formatted at 2026-09-29 03:16:13 on dist-test-slave-nbjg"
server_key: "4adde61997e5ca456f86c16f81eda154"
server_key_iv: "ac7d1bd6d45c4d52cf5b9332f10634fa"
server_key_version: "encryptionkey@0"
I20260929 03:16:13.306437 24580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:16:13.308861 24595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:13.310124 24580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:16:13.310277 24580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "463753b4f63d4e0c9b555f21518294ff"
format_stamp: "Formatted at 2026-09-29 03:16:13 on dist-test-slave-nbjg"
server_key: "4adde61997e5ca456f86c16f81eda154"
server_key_iv: "ac7d1bd6d45c4d52cf5b9332f10634fa"
server_key_version: "encryptionkey@0"
I20260929 03:16:13.310379 24580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:16:13.334578 24580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:16:13.337850 24580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:16:13.338033 24580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:16:13.338788 24580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:16:13.339763 24580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:16:13.339811 24580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:13.339851 24580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:16:13.339871 24580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:13.348073 24580 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:39107
I20260929 03:16:13.348098 24708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:39107 every 8 connection(s)
I20260929 03:16:13.348965 24580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
Sep 29 03:16:13 dist-test-slave-nbjg krb5kdc[24473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651773, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:16:13.358888 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 24580
I20260929 03:16:13.359001 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance
I20260929 03:16:13.359266 14266 external_mini_cluster.cc:1428] Setting key 60f7cc33bdcfe06f45aceb45abc78b7e
I20260929 03:16:13.362720 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:13.350731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:44925 (local address 127.13.238.190:39579)
0929 03:16:13.350880 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:13.350885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:13.351803 (+ 918us) server_negotiation.cc:407] Connection header received
0929 03:16:13.353135 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:13.353138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:13.353188 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:13.353263 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:13.354520 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.355070 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.356275 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.356494 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.356560 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:13.359114 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:16:13.359117 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:16:13.359119 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:16:13.359145 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:16:13.360516 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:16:13.361100 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:16:13.361103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:16:13.361105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:16:13.361200 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:16:13.361553 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:16:13.361555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:16:13.361556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:16:13.361732 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:16:13.361803 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:16:13.362421 (+ 618us) server_negotiation.cc:299] Negotiation successful
0929 03:16:13.362543 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:16:13.363653 24711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:13.351132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:39579 (local address 127.13.238.129:44925)
0929 03:16:13.351600 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:13.351644 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:13.352923 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:13.353417 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:13.353425 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:13.353824 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:13.354381 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:13.354398 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.355206 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.355209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:13.356169 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:13.356179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.356621 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.356625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:13.356778 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:13.357302 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:13.357320 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:13.358952 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:16:13.360628 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:16:13.360633 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:16:13.360648 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:16:13.361008 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:16:13.361284 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:16:13.361287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:16:13.361289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:16:13.361479 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:16:13.361815 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:16:13.361823 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:16:13.362143 (+ 320us) client_negotiation.cc:769] Sending connection context
0929 03:16:13.362381 (+ 238us) client_negotiation.cc:240] Negotiation successful
0929 03:16:13.362768 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":107,"threads_started":1}
I20260929 03:16:13.364897 24709 heartbeater.cc:347] Connected to a master server at 127.13.238.190:39579
I20260929 03:16:13.365135 24709 heartbeater.cc:464] Registering TS with master...
I20260929 03:16:13.365618 24709 heartbeater.cc:511] Master 127.13.238.190:39579 requested a full tablet report, sending...
I20260929 03:16:13.367136 24518 ts_manager.cc:194] Registered new tserver with Master: 463753b4f63d4e0c9b555f21518294ff (127.13.238.129:39107)
I20260929 03:16:13.368249 24518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.129@KRBTEST.COM'} at 127.13.238.129:44925
WARNING: no policy specified for kudu/127.13.238.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
I20260929 03:16:13.385298 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:0
--local_ip_for_outbound_sockets=127.13.238.130
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:39579
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:16:13.496991 24714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:16:13.497267 24714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:16:13.497310 24714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:16:13.501540 24714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:16:13.501605 24714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:16:13.501664 24714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:16:13.507030 24714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:39579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:16:13.508081 24714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:16:13.509071 24714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:16:13.518604 24722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:16:13.518738 24714 server_base.cc:1034] running on GCE node
W20260929 03:16:13.518610 24720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:16:13.518683 24719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:16:13.519397 24714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:16:13.520023 24714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:16:13.521205 24714 hybrid_clock.cc:648] HybridClock initialized: now 1790651773521187 us; error 35 us; skew 500 ppm
Sep 29 03:16:13 dist-test-slave-nbjg krb5kdc[24473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651773, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:16:13.523890 24714 init.cc:377] Logged in from keytab as kudu/127.13.238.130@KRBTEST.COM (short username kudu)
I20260929 03:16:13.524742 24714 webserver.cc:460] Webserver started at http://127.13.238.130:46607/ using document root <none> and password file <none>
I20260929 03:16:13.525277 24714 fs_manager.cc:359] Metadata directory not provided
I20260929 03:16:13.525334 24714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:16:13.525497 24714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:16:13.527529 24714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/data/instance:
uuid: "2b4db34f22864fadb44898d085e1c7cd"
format_stamp: "Formatted at 2026-09-29 03:16:13 on dist-test-slave-nbjg"
server_key: "5efb59a75bfb48c60246af29f5ec832f"
server_key_iv: "ccdab232aa9d9b39f4f5b8cdce729326"
server_key_version: "encryptionkey@0"
I20260929 03:16:13.528030 24714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance:
uuid: "2b4db34f22864fadb44898d085e1c7cd"
format_stamp: "Formatted at 2026-09-29 03:16:13 on dist-test-slave-nbjg"
server_key: "5efb59a75bfb48c60246af29f5ec832f"
server_key_iv: "ccdab232aa9d9b39f4f5b8cdce729326"
server_key_version: "encryptionkey@0"
I20260929 03:16:13.531620 24714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:16:13.533977 24729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:13.535243 24714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:16:13.535352 24714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "2b4db34f22864fadb44898d085e1c7cd"
format_stamp: "Formatted at 2026-09-29 03:16:13 on dist-test-slave-nbjg"
server_key: "5efb59a75bfb48c60246af29f5ec832f"
server_key_iv: "ccdab232aa9d9b39f4f5b8cdce729326"
server_key_version: "encryptionkey@0"
I20260929 03:16:13.535449 24714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:16:13.564277 24714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:16:13.567700 24714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:16:13.567893 24714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:16:13.568523 24714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:16:13.569402 24714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:16:13.569448 24714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:13.569504 24714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:16:13.569538 24714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:13.578052 24714 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:37291
I20260929 03:16:13.578080 24842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:37291 every 8 connection(s)
I20260929 03:16:13.578990 24714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:16:13.582104 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 24714
I20260929 03:16:13.582248 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance
I20260929 03:16:13.582573 14266 external_mini_cluster.cc:1428] Setting key 74d1738d71d162ec286c8503dfc6a905
Sep 29 03:16:13 dist-test-slave-nbjg krb5kdc[24473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651773, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.238.131@KRBTEST.COM; defaulting to no policy
I20260929 03:16:13.593974 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:13.580893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:59823 (local address 127.13.238.190:39579)
0929 03:16:13.581043 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:13.581047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:13.581920 (+ 873us) server_negotiation.cc:407] Connection header received
0929 03:16:13.583634 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:13.583638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:13.583683 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:13.583768 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:13.585307 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.585824 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.587474 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.587705 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.587816 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:13.590573 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:16:13.590576 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:16:13.590578 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:16:13.590603 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:16:13.591970 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:16:13.592487 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:16:13.592490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:16:13.592491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:16:13.592563 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:16:13.592884 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:16:13.592886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:16:13.592888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:16:13.593092 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:16:13.593176 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 03:16:13.593685 (+ 509us) server_negotiation.cc:299] Negotiation successful
0929 03:16:13.593811 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.131@KRBTEST.COM" created.
I20260929 03:16:13.595031 24845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:13.581277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:39579 (local address 127.13.238.130:59823)
0929 03:16:13.581746 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:13.581790 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:13.583433 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:13.583950 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:13.583961 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:13.584444 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:13.585161 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:13.585180 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.585946 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.585949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:13.587366 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:13.587377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.587853 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.587856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:13.588071 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:13.588668 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:13.588689 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:13.590401 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:16:13.592076 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:16:13.592083 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:16:13.592101 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:16:13.592387 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:16:13.592641 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:16:13.592644 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:16:13.592646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:16:13.592789 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:16:13.593185 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:16:13.593192 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:16:13.593447 (+ 255us) client_negotiation.cc:769] Sending connection context
0929 03:16:13.593612 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 03:16:13.593989 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":114,"threads_started":1}
I20260929 03:16:13.596328 24843 heartbeater.cc:347] Connected to a master server at 127.13.238.190:39579
I20260929 03:16:13.596654 24843 heartbeater.cc:464] Registering TS with master...
I20260929 03:16:13.597249 24843 heartbeater.cc:511] Master 127.13.238.190:39579 requested a full tablet report, sending...
I20260929 03:16:13.598325 24518 ts_manager.cc:194] Registered new tserver with Master: 2b4db34f22864fadb44898d085e1c7cd (127.13.238.130:37291)
I20260929 03:16:13.598939 24518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.130@KRBTEST.COM'} at 127.13.238.130:59823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
I20260929 03:16:13.609249 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:0
--local_ip_for_outbound_sockets=127.13.238.131
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:39579
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:16:13.719369 24848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:16:13.719651 24848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:16:13.719712 24848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:16:13.723809 24848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:16:13.723873 24848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:16:13.723950 24848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:16:13.729375 24848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:39579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:16:13.730522 24848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:16:13.731570 24848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:16:13.741394 24853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:16:13.741381 24854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:16:13.741400 24856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:16:13.741446 24848 server_base.cc:1034] running on GCE node
I20260929 03:16:13.742024 24848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:16:13.742720 24848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:16:13.743883 24848 hybrid_clock.cc:648] HybridClock initialized: now 1790651773743862 us; error 32 us; skew 500 ppm
Sep 29 03:16:13 dist-test-slave-nbjg krb5kdc[24473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651773, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:16:13.746486 24848 init.cc:377] Logged in from keytab as kudu/127.13.238.131@KRBTEST.COM (short username kudu)
I20260929 03:16:13.747376 24848 webserver.cc:460] Webserver started at http://127.13.238.131:38347/ using document root <none> and password file <none>
I20260929 03:16:13.747929 24848 fs_manager.cc:359] Metadata directory not provided
I20260929 03:16:13.748003 24848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:16:13.748214 24848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:16:13.750293 24848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/data/instance:
uuid: "065cd60bcc0a4904a00598b2a8514eac"
format_stamp: "Formatted at 2026-09-29 03:16:13 on dist-test-slave-nbjg"
server_key: "b062fb6304cb45531aba01b517cfa4fa"
server_key_iv: "a2c89ad8ae7db170dc3cf004e7eabab2"
server_key_version: "encryptionkey@0"
I20260929 03:16:13.750836 24848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance:
uuid: "065cd60bcc0a4904a00598b2a8514eac"
format_stamp: "Formatted at 2026-09-29 03:16:13 on dist-test-slave-nbjg"
server_key: "b062fb6304cb45531aba01b517cfa4fa"
server_key_iv: "a2c89ad8ae7db170dc3cf004e7eabab2"
server_key_version: "encryptionkey@0"
I20260929 03:16:13.754720 24848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 03:16:13.757375 24863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:13.758754 24848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:16:13.758876 24848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "065cd60bcc0a4904a00598b2a8514eac"
format_stamp: "Formatted at 2026-09-29 03:16:13 on dist-test-slave-nbjg"
server_key: "b062fb6304cb45531aba01b517cfa4fa"
server_key_iv: "a2c89ad8ae7db170dc3cf004e7eabab2"
server_key_version: "encryptionkey@0"
I20260929 03:16:13.758983 24848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:16:13.771641 24848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:16:13.774724 24848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:16:13.774888 24848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:16:13.775458 24848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:16:13.776269 24848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:16:13.776314 24848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:13.776367 24848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:16:13.776396 24848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:13.785041 24848 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:45859
I20260929 03:16:13.785087 24976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:45859 every 8 connection(s)
I20260929 03:16:13.785955 24848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
Sep 29 03:16:13 dist-test-slave-nbjg krb5kdc[24473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651773, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:16:13.795323 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 24848
I20260929 03:16:13.795449 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance
I20260929 03:16:13.795789 14266 external_mini_cluster.cc:1428] Setting key 9a48d1492ee16f7930902b9f3de58ed0
I20260929 03:16:13.799472 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:13.787774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:32875 (local address 127.13.238.190:39579)
0929 03:16:13.787906 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:13.787910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:13.788844 (+ 934us) server_negotiation.cc:407] Connection header received
0929 03:16:13.790217 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:13.790221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:13.790278 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:13.790353 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:13.791523 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.792047 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.793145 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.793357 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.793429 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:13.795964 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:16:13.795967 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:16:13.795969 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:16:13.795998 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:16:13.797481 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:16:13.798133 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:16:13.798145 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:16:13.798147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:16:13.798273 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:16:13.798554 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:16:13.798556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:16:13.798557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:16:13.798734 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:16:13.798805 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:16:13.799242 (+ 437us) server_negotiation.cc:299] Negotiation successful
0929 03:16:13.799347 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:16:13.800336 24979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:13.788199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:39579 (local address 127.13.238.131:32875)
0929 03:16:13.788672 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:13.788737 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:13.789992 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:13.790513 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:13.790520 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:13.790901 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:13.791402 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:13.791417 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.792141 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.792143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:13.793044 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:13.793053 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.793473 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.793476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:13.793620 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:13.794161 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:13.794179 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:13.795753 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:16:13.797602 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:16:13.797609 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:16:13.797626 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:16:13.798024 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:16:13.798345 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:16:13.798347 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:16:13.798348 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:16:13.798486 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:16:13.798806 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:16:13.798811 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:16:13.799056 (+ 245us) client_negotiation.cc:769] Sending connection context
0929 03:16:13.799215 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 03:16:13.799486 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":127,"threads_started":1}
I20260929 03:16:13.801393 24977 heartbeater.cc:347] Connected to a master server at 127.13.238.190:39579
I20260929 03:16:13.801628 24977 heartbeater.cc:464] Registering TS with master...
I20260929 03:16:13.802107 24977 heartbeater.cc:511] Master 127.13.238.190:39579 requested a full tablet report, sending...
I20260929 03:16:13.803104 24518 ts_manager.cc:194] Registered new tserver with Master: 065cd60bcc0a4904a00598b2a8514eac (127.13.238.131:45859)
I20260929 03:16:13.803642 24518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.131@KRBTEST.COM'} at 127.13.238.131:32875
I20260929 03:16:13.809609 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:16:13.818361 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:13.811468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41712 (local address 127.13.238.190:39579)
0929 03:16:13.811597 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:13.811601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:13.811754 (+ 153us) server_negotiation.cc:407] Connection header received
0929 03:16:13.811871 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:13.811875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:13.811919 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:13.811981 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:13.812662 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.813186 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.814006 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.814270 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.814358 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:13.815613 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:16:13.815617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:16:13.815619 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:16:13.815643 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:16:13.816837 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:16:13.817309 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:16:13.817312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:16:13.817313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:16:13.817393 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:16:13.817610 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:16:13.817612 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:16:13.817613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:16:13.817822 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:16:13.817922 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 03:16:13.818090 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 03:16:13.818183 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 03:16:13.821002 24518 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:16:13.824092 24518 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:16:13.837997 24990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:13.832558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.13.238.131:45859)
0929 03:16:13.833757 (+ 1199us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:13.833770 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:13.833794 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:16:13.833871 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:13.833876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:13.834084 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:13.834265 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:16:13.834554 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.835337 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.836777 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.837325 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.837432 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:13.837590 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 03:16:13.837751 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 03:16:13.837845 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1090,"thread_start_us":132,"threads_started":1}
I20260929 03:16:13.837990 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:13.832458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:45859 (local address 127.0.0.1:43136)
0929 03:16:13.832918 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:13.832964 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:13.833084 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:13.834262 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:13.834269 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:13.834293 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:16:13.834443 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:13.834451 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.835455 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.835459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:13.836659 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:13.836668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.837431 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.837435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:13.837560 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:13.837600 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 03:16:13.837711 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 03:16:13.837837 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":24,"thread_start_us":84,"threads_started":1}
I20260929 03:16:13.839043 24988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:13.832557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:39107 (local address 127.0.0.1:48942)
0929 03:16:13.833472 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:13.833485 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:13.833557 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:13.834437 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:13.834439 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:13.834455 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:16:13.834541 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:13.834547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.835986 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.835989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:13.837361 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:13.837372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.838633 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.838637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:13.838734 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:13.838757 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:16:13.838824 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 03:16:13.838911 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":106,"thread_start_us":122,"threads_started":1}
I20260929 03:16:13.839202 24991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:13.833184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:37291 (local address 127.0.0.1:38962)
0929 03:16:13.833769 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:13.833782 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:13.833866 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:13.834486 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:13.834489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:13.834506 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:16:13.834581 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:13.834586 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.835986 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.835989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:13.837362 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:13.837372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.838781 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.838785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:13.838901 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:13.838924 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:16:13.838991 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 03:16:13.839066 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":126,"thread_start_us":99,"threads_started":1}
I20260929 03:16:13.839468 24993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:13.833412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48942 (local address 127.13.238.129:39107)
0929 03:16:13.833830 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:13.833842 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:13.833860 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:16:13.833917 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:13.833921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:13.834084 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:13.834265 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:16:13.834641 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.835833 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.837514 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.838405 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.838520 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:13.838648 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 03:16:13.839214 (+ 566us) server_negotiation.cc:299] Negotiation successful
0929 03:16:13.839320 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":106,"threads_started":1}
I20260929 03:16:13.841110 24911 tablet_service.cc:1467] Processing CreateTablet for tablet a3b8792e979b4b65ad5da3b5fa05283f (DEFAULT_TABLE table=test-table [id=89fe6e19a3bd4c9c97e819d6705a03b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:16:13.842089 24911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3b8792e979b4b65ad5da3b5fa05283f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:16:13.842892 24992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:13.833392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38962 (local address 127.13.238.130:37291)
0929 03:16:13.833807 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:13.833818 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:13.833840 (+ 22us) server_negotiation.cc:407] Connection header received
0929 03:16:13.833989 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:13.833995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:13.834147 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:13.834345 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:16:13.834704 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.835833 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.837514 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:13.838657 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:13.838772 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:13.838892 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 03:16:13.842631 (+ 3739us) server_negotiation.cc:299] Negotiation successful
0929 03:16:13.842761 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":124,"threads_started":1}
I20260929 03:16:13.845425 24777 tablet_service.cc:1467] Processing CreateTablet for tablet a3b8792e979b4b65ad5da3b5fa05283f (DEFAULT_TABLE table=test-table [id=89fe6e19a3bd4c9c97e819d6705a03b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:16:13.845422 24643 tablet_service.cc:1467] Processing CreateTablet for tablet a3b8792e979b4b65ad5da3b5fa05283f (DEFAULT_TABLE table=test-table [id=89fe6e19a3bd4c9c97e819d6705a03b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:16:13.846414 24777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3b8792e979b4b65ad5da3b5fa05283f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:16:13.846504 24643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3b8792e979b4b65ad5da3b5fa05283f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:16:13.850376 24994 tablet_bootstrap.cc:492] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac: Bootstrap starting.
I20260929 03:16:13.852821 24994 tablet_bootstrap.cc:654] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac: Neither blocks nor log segments found. Creating new log.
I20260929 03:16:13.853300 24996 tablet_bootstrap.cc:492] T a3b8792e979b4b65ad5da3b5fa05283f P 463753b4f63d4e0c9b555f21518294ff: Bootstrap starting.
I20260929 03:16:13.853741 24995 tablet_bootstrap.cc:492] T a3b8792e979b4b65ad5da3b5fa05283f P 2b4db34f22864fadb44898d085e1c7cd: Bootstrap starting.
I20260929 03:16:13.854606 24994 log.cc:824] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac: Log is configured to *not* fsync() on all Append() calls
I20260929 03:16:13.855902 24996 tablet_bootstrap.cc:654] T a3b8792e979b4b65ad5da3b5fa05283f P 463753b4f63d4e0c9b555f21518294ff: Neither blocks nor log segments found. Creating new log.
I20260929 03:16:13.856774 24994 tablet_bootstrap.cc:492] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac: No bootstrap required, opened a new log
I20260929 03:16:13.856932 24996 log.cc:824] T a3b8792e979b4b65ad5da3b5fa05283f P 463753b4f63d4e0c9b555f21518294ff: Log is configured to *not* fsync() on all Append() calls
I20260929 03:16:13.856984 24994 ts_tablet_manager.cc:1397] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 03:16:13.857350 24995 tablet_bootstrap.cc:654] T a3b8792e979b4b65ad5da3b5fa05283f P 2b4db34f22864fadb44898d085e1c7cd: Neither blocks nor log segments found. Creating new log.
I20260929 03:16:13.858178 24995 log.cc:824] T a3b8792e979b4b65ad5da3b5fa05283f P 2b4db34f22864fadb44898d085e1c7cd: Log is configured to *not* fsync() on all Append() calls
I20260929 03:16:13.858871 24996 tablet_bootstrap.cc:492] T a3b8792e979b4b65ad5da3b5fa05283f P 463753b4f63d4e0c9b555f21518294ff: No bootstrap required, opened a new log
I20260929 03:16:13.859081 24996 ts_tablet_manager.cc:1397] T a3b8792e979b4b65ad5da3b5fa05283f P 463753b4f63d4e0c9b555f21518294ff: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 03:16:13.860267 24995 tablet_bootstrap.cc:492] T a3b8792e979b4b65ad5da3b5fa05283f P 2b4db34f22864fadb44898d085e1c7cd: No bootstrap required, opened a new log
I20260929 03:16:13.860476 24995 ts_tablet_manager.cc:1397] T a3b8792e979b4b65ad5da3b5fa05283f P 2b4db34f22864fadb44898d085e1c7cd: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260929 03:16:13.860394 24994 raft_consensus.cc:348] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065cd60bcc0a4904a00598b2a8514eac" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 45859 } } peers { permanent_uuid: "463753b4f63d4e0c9b555f21518294ff" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39107 } } peers { permanent_uuid: "2b4db34f22864fadb44898d085e1c7cd" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37291 } }
I20260929 03:16:13.860663 24994 raft_consensus.cc:374] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:16:13.860714 24994 raft_consensus.cc:729] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 065cd60bcc0a4904a00598b2a8514eac, State: Initialized, Role: FOLLOWER
I20260929 03:16:13.861235 24994 consensus_queue.cc:260] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065cd60bcc0a4904a00598b2a8514eac" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 45859 } } peers { permanent_uuid: "463753b4f63d4e0c9b555f21518294ff" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39107 } } peers { permanent_uuid: "2b4db34f22864fadb44898d085e1c7cd" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37291 } }
I20260929 03:16:13.862026 24977 heartbeater.cc:503] Master 127.13.238.190:39579 was elected leader, sending a full tablet report...
I20260929 03:16:13.862319 24994 ts_tablet_manager.cc:1428] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260929 03:16:13.863092 24996 raft_consensus.cc:348] T a3b8792e979b4b65ad5da3b5fa05283f P 463753b4f63d4e0c9b555f21518294ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065cd60bcc0a4904a00598b2a8514eac" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 45859 } } peers { permanent_uuid: "463753b4f63d4e0c9b555f21518294ff" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39107 } } peers { permanent_uuid: "2b4db34f22864fadb44898d085e1c7cd" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37291 } }
I20260929 03:16:13.863360 24996 raft_consensus.cc:374] T a3b8792e979b4b65ad5da3b5fa05283f P 463753b4f63d4e0c9b555f21518294ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:16:13.863416 24996 raft_consensus.cc:729] T a3b8792e979b4b65ad5da3b5fa05283f P 463753b4f63d4e0c9b555f21518294ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 463753b4f63d4e0c9b555f21518294ff, State: Initialized, Role: FOLLOWER
I20260929 03:16:13.863886 24996 consensus_queue.cc:260] T a3b8792e979b4b65ad5da3b5fa05283f P 463753b4f63d4e0c9b555f21518294ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065cd60bcc0a4904a00598b2a8514eac" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 45859 } } peers { permanent_uuid: "463753b4f63d4e0c9b555f21518294ff" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39107 } } peers { permanent_uuid: "2b4db34f22864fadb44898d085e1c7cd" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37291 } }
I20260929 03:16:13.864921 24996 ts_tablet_manager.cc:1428] T a3b8792e979b4b65ad5da3b5fa05283f P 463753b4f63d4e0c9b555f21518294ff: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 03:16:13.865402 24709 heartbeater.cc:503] Master 127.13.238.190:39579 was elected leader, sending a full tablet report...
I20260929 03:16:13.865612 24995 raft_consensus.cc:348] T a3b8792e979b4b65ad5da3b5fa05283f P 2b4db34f22864fadb44898d085e1c7cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065cd60bcc0a4904a00598b2a8514eac" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 45859 } } peers { permanent_uuid: "463753b4f63d4e0c9b555f21518294ff" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39107 } } peers { permanent_uuid: "2b4db34f22864fadb44898d085e1c7cd" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37291 } }
I20260929 03:16:13.865885 24995 raft_consensus.cc:374] T a3b8792e979b4b65ad5da3b5fa05283f P 2b4db34f22864fadb44898d085e1c7cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:16:13.865948 24995 raft_consensus.cc:729] T a3b8792e979b4b65ad5da3b5fa05283f P 2b4db34f22864fadb44898d085e1c7cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b4db34f22864fadb44898d085e1c7cd, State: Initialized, Role: FOLLOWER
I20260929 03:16:13.866451 24995 consensus_queue.cc:260] T a3b8792e979b4b65ad5da3b5fa05283f P 2b4db34f22864fadb44898d085e1c7cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065cd60bcc0a4904a00598b2a8514eac" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 45859 } } peers { permanent_uuid: "463753b4f63d4e0c9b555f21518294ff" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39107 } } peers { permanent_uuid: "2b4db34f22864fadb44898d085e1c7cd" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37291 } }
I20260929 03:16:13.867233 24843 heartbeater.cc:503] Master 127.13.238.190:39579 was elected leader, sending a full tablet report...
I20260929 03:16:13.867452 24995 ts_tablet_manager.cc:1428] T a3b8792e979b4b65ad5da3b5fa05283f P 2b4db34f22864fadb44898d085e1c7cd: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260929 03:16:14.020808 25000 raft_consensus.cc:482] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:16:14.021080 25000 raft_consensus.cc:504] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065cd60bcc0a4904a00598b2a8514eac" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 45859 } } peers { permanent_uuid: "463753b4f63d4e0c9b555f21518294ff" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39107 } } peers { permanent_uuid: "2b4db34f22864fadb44898d085e1c7cd" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37291 } }
I20260929 03:16:14.022326 25000 leader_election.cc:288] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 463753b4f63d4e0c9b555f21518294ff (127.13.238.129:39107), 2b4db34f22864fadb44898d085e1c7cd (127.13.238.130:37291)
I20260929 03:16:14.026969 24992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:14.022727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:46315 (local address 127.13.238.130:37291)
0929 03:16:14.022865 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:14.022869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:14.023201 (+ 332us) server_negotiation.cc:407] Connection header received
0929 03:16:14.023273 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:14.023275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:14.023314 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:14.023384 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:16:14.023599 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:14.024290 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:14.025705 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:14.026328 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:14.026444 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:14.026496 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:16:14.026769 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 03:16:14.026836 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:16:14.026969 24993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:14.022662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:58569 (local address 127.13.238.129:39107)
0929 03:16:14.022799 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:14.022804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:14.022821 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:16:14.022972 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:14.022975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:14.023027 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:14.023093 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:16:14.023370 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:14.024113 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:14.025698 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:14.026329 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:14.026444 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:14.026496 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:16:14.026769 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 03:16:14.026836 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 03:16:14.027005 24979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:14.022570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:39107 (local address 127.13.238.131:58569)
0929 03:16:14.022739 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:14.022755 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:14.022857 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:14.023122 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:14.023125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:14.023146 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:16:14.023275 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:14.023282 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:14.024216 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:14.024219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:14.025571 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:14.025590 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:14.026463 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:14.026467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:14.026655 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:14.026674 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 03:16:14.026752 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:16:14.026855 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":112}
I20260929 03:16:14.027014 25003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:14.022570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:37291 (local address 127.13.238.131:46315)
0929 03:16:14.023104 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:14.023117 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:14.023200 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:14.023404 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:14.023406 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:14.023420 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:16:14.023504 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:14.023508 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:14.024385 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:14.024388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:14.025571 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:14.025583 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:14.026463 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:14.026467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:14.026633 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:14.026671 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 03:16:14.026752 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 03:16:14.026855 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":276,"thread_start_us":104,"threads_started":1}
I20260929 03:16:14.027724 24663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b8792e979b4b65ad5da3b5fa05283f" candidate_uuid: "065cd60bcc0a4904a00598b2a8514eac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "463753b4f63d4e0c9b555f21518294ff" is_pre_election: true
I20260929 03:16:14.027747 24797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b8792e979b4b65ad5da3b5fa05283f" candidate_uuid: "065cd60bcc0a4904a00598b2a8514eac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b4db34f22864fadb44898d085e1c7cd" is_pre_election: true
I20260929 03:16:14.028007 24797 raft_consensus.cc:2454] T a3b8792e979b4b65ad5da3b5fa05283f P 2b4db34f22864fadb44898d085e1c7cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 065cd60bcc0a4904a00598b2a8514eac in term 0.
I20260929 03:16:14.028004 24663 raft_consensus.cc:2454] T a3b8792e979b4b65ad5da3b5fa05283f P 463753b4f63d4e0c9b555f21518294ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 065cd60bcc0a4904a00598b2a8514eac in term 0.
I20260929 03:16:14.028450 24865 leader_election.cc:302] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 065cd60bcc0a4904a00598b2a8514eac, 2b4db34f22864fadb44898d085e1c7cd, 463753b4f63d4e0c9b555f21518294ff; no voters:
I20260929 03:16:14.028719 25000 raft_consensus.cc:2781] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:16:14.028800 25000 raft_consensus.cc:482] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:16:14.028844 25000 raft_consensus.cc:3037] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:16:14.029806 25000 raft_consensus.cc:504] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065cd60bcc0a4904a00598b2a8514eac" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 45859 } } peers { permanent_uuid: "463753b4f63d4e0c9b555f21518294ff" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39107 } } peers { permanent_uuid: "2b4db34f22864fadb44898d085e1c7cd" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37291 } }
I20260929 03:16:14.030104 25000 leader_election.cc:288] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac [CANDIDATE]: Term 1 election: Requested vote from peers 463753b4f63d4e0c9b555f21518294ff (127.13.238.129:39107), 2b4db34f22864fadb44898d085e1c7cd (127.13.238.130:37291)
I20260929 03:16:14.030478 24797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b8792e979b4b65ad5da3b5fa05283f" candidate_uuid: "065cd60bcc0a4904a00598b2a8514eac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b4db34f22864fadb44898d085e1c7cd"
I20260929 03:16:14.030455 24663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b8792e979b4b65ad5da3b5fa05283f" candidate_uuid: "065cd60bcc0a4904a00598b2a8514eac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "463753b4f63d4e0c9b555f21518294ff"
I20260929 03:16:14.030577 24797 raft_consensus.cc:3037] T a3b8792e979b4b65ad5da3b5fa05283f P 2b4db34f22864fadb44898d085e1c7cd [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:16:14.030598 24663 raft_consensus.cc:3037] T a3b8792e979b4b65ad5da3b5fa05283f P 463753b4f63d4e0c9b555f21518294ff [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:16:14.031549 24797 raft_consensus.cc:2454] T a3b8792e979b4b65ad5da3b5fa05283f P 2b4db34f22864fadb44898d085e1c7cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 065cd60bcc0a4904a00598b2a8514eac in term 1.
I20260929 03:16:14.031569 24663 raft_consensus.cc:2454] T a3b8792e979b4b65ad5da3b5fa05283f P 463753b4f63d4e0c9b555f21518294ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 065cd60bcc0a4904a00598b2a8514eac in term 1.
I20260929 03:16:14.031785 24865 leader_election.cc:302] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 065cd60bcc0a4904a00598b2a8514eac, 2b4db34f22864fadb44898d085e1c7cd; no voters:
I20260929 03:16:14.031941 25000 raft_consensus.cc:2781] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:16:14.032145 25000 raft_consensus.cc:686] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac [term 1 LEADER]: Becoming Leader. State: Replica: 065cd60bcc0a4904a00598b2a8514eac, State: Running, Role: LEADER
I20260929 03:16:14.032440 25000 consensus_queue.cc:237] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065cd60bcc0a4904a00598b2a8514eac" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 45859 } } peers { permanent_uuid: "463753b4f63d4e0c9b555f21518294ff" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39107 } } peers { permanent_uuid: "2b4db34f22864fadb44898d085e1c7cd" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37291 } }
I20260929 03:16:14.035485 24517 catalog_manager.cc:5495] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac reported cstate change: term changed from 0 to 1, leader changed from <none> to 065cd60bcc0a4904a00598b2a8514eac (127.13.238.131). New cstate: current_term: 1 leader_uuid: "065cd60bcc0a4904a00598b2a8514eac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "065cd60bcc0a4904a00598b2a8514eac" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 45859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "463753b4f63d4e0c9b555f21518294ff" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b4db34f22864fadb44898d085e1c7cd" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37291 } health_report { overall_health: UNKNOWN } } }
W20260929 03:16:14.036867 24978 tablet.cc:2369] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:16:14.072602 24990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:14.069146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43146 (local address 127.13.238.131:45859)
0929 03:16:14.069289 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:14.069293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:14.069310 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:16:14.069391 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:14.069394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:14.069454 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:14.069539 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:16:14.069795 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:14.070330 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:14.071185 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:14.071407 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:14.071493 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:14.071787 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:16:14.072232 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:16:14.072336 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 03:16:14.072402 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 03:16:14.072463 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 03:16:14.078290 24663 raft_consensus.cc:1261] T a3b8792e979b4b65ad5da3b5fa05283f P 463753b4f63d4e0c9b555f21518294ff [term 1 FOLLOWER]: Refusing update from remote peer 065cd60bcc0a4904a00598b2a8514eac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:16:14.078291 24797 raft_consensus.cc:1261] T a3b8792e979b4b65ad5da3b5fa05283f P 2b4db34f22864fadb44898d085e1c7cd [term 1 FOLLOWER]: Refusing update from remote peer 065cd60bcc0a4904a00598b2a8514eac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:16:14.078949 25000 consensus_queue.cc:1035] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac [LEADER]: Connected to new peer: Peer: permanent_uuid: "463753b4f63d4e0c9b555f21518294ff" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:16:14.079124 25004 consensus_queue.cc:1035] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b4db34f22864fadb44898d085e1c7cd" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 03:16:14.080051 24844 tablet.cc:2369] T a3b8792e979b4b65ad5da3b5fa05283f P 2b4db34f22864fadb44898d085e1c7cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:16:14.085549 25007 mvcc.cc:205] Tried to move back new op lower bound from 7334509666617511936 to 7334509666439344128. Current Snapshot: MvccSnapshot[applied={T|T < 7334509666617511936}]
I20260929 03:16:14.086585 25009 mvcc.cc:205] Tried to move back new op lower bound from 7334509666617511936 to 7334509666439344128. Current Snapshot: MvccSnapshot[applied={T|T < 7334509666617511936}]
I20260929 03:16:14.087656 25008 mvcc.cc:205] Tried to move back new op lower bound from 7334509666617511936 to 7334509666439344128. Current Snapshot: MvccSnapshot[applied={T|T < 7334509666617511936}]
I20260929 03:16:14.094693 24517 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:16:14.094884 24517 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:16:14.097434 24517 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 257cf5367b2a484da6e660454cbe596d: Sending DeleteTablet for 3 replicas of tablet a3b8792e979b4b65ad5da3b5fa05283f
I20260929 03:16:14.098285 24911 tablet_service.cc:1514] Processing DeleteTablet for tablet a3b8792e979b4b65ad5da3b5fa05283f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:16:14 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:43136
I20260929 03:16:14.098289 24643 tablet_service.cc:1514] Processing DeleteTablet for tablet a3b8792e979b4b65ad5da3b5fa05283f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:16:14 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:48942
I20260929 03:16:14.098415 24777 tablet_service.cc:1514] Processing DeleteTablet for tablet a3b8792e979b4b65ad5da3b5fa05283f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:16:14 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:38962
I20260929 03:16:14.098807 25018 tablet_replica.cc:330] T a3b8792e979b4b65ad5da3b5fa05283f P 463753b4f63d4e0c9b555f21518294ff: stopping tablet replica
I20260929 03:16:14.099139 25018 raft_consensus.cc:2229] T a3b8792e979b4b65ad5da3b5fa05283f P 463753b4f63d4e0c9b555f21518294ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:16:14.099443 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 24580
I20260929 03:16:14.103397 25016 tablet_replica.cc:330] T a3b8792e979b4b65ad5da3b5fa05283f P 2b4db34f22864fadb44898d085e1c7cd: stopping tablet replica
I20260929 03:16:14.103746 25016 raft_consensus.cc:2229] T a3b8792e979b4b65ad5da3b5fa05283f P 2b4db34f22864fadb44898d085e1c7cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:16:14.104099 25016 raft_consensus.cc:2258] T a3b8792e979b4b65ad5da3b5fa05283f P 2b4db34f22864fadb44898d085e1c7cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:16:14.104900 25017 tablet_replica.cc:330] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac: stopping tablet replica
I20260929 03:16:14.105206 25017 raft_consensus.cc:2229] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:16:14.105554 25017 raft_consensus.cc:2258] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:16:14.105814 25016 ts_tablet_manager.cc:1905] T a3b8792e979b4b65ad5da3b5fa05283f P 2b4db34f22864fadb44898d085e1c7cd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 03:16:14.106513 24505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.238.129:39107 (error 108)
I20260929 03:16:14.107298 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 24714
I20260929 03:16:14.107456 25017 ts_tablet_manager.cc:1905] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:16:14.110271 25017 ts_tablet_manager.cc:1918] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:16:14.110368 25017 log.cc:1196] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/a3b8792e979b4b65ad5da3b5fa05283f
I20260929 03:16:14.110677 25017 ts_tablet_manager.cc:1939] T a3b8792e979b4b65ad5da3b5fa05283f P 065cd60bcc0a4904a00598b2a8514eac: Deleting consensus metadata
I20260929 03:16:14.111945 24503 catalog_manager.cc:4841] TS 065cd60bcc0a4904a00598b2a8514eac (127.13.238.131:45859): tablet a3b8792e979b4b65ad5da3b5fa05283f (table test-table [id=89fe6e19a3bd4c9c97e819d6705a03b2]) successfully deleted
I20260929 03:16:14.112533 24991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:14.112254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:39107 (local address 127.0.0.1:48958)
0929 03:16:14.112378 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:14.112447 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.129:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 03:16:14.112700 24505 catalog_manager.cc:4584] TS 463753b4f63d4e0c9b555f21518294ff (127.13.238.129:39107): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a3b8792e979b4b65ad5da3b5fa05283f: Network error: Client connection negotiation failed: client connection to 127.13.238.129:39107: connect: Connection refused (error 111)
I20260929 03:16:14.113976 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 24848
I20260929 03:16:14.116182 24991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:14.114131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:37291 (local address 127.0.0.1:38966)
0929 03:16:14.116057 (+ 1926us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:14.116109 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.130:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1848}
I20260929 03:16:14.120194 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 24486
2026-09-29T03:16:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2306 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:16:14 dist-test-slave-nbjg krb5kdc[25022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:16:14 dist-test-slave-nbjg krb5kdc[25022](info): set up 2 sockets
Sep 29 03:16:14 dist-test-slave-nbjg krb5kdc[25022](info): commencing operation
krb5kdc: starting...
W20260929 03:16:15.079116 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:16:15 dist-test-slave-nbjg krb5kdc[25022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:16:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:16:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.13.238.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.13.238.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.13.238.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/rangeradmin_127.13.238.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.13.238.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/rangeradmin_127.13.238.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.13.238.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.13.238.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.13.238.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/rangerlookup_127.13.238.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.13.238.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/rangerlookup_127.13.238.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.238.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.238.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.238.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/HTTP_127.13.238.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.238.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/HTTP_127.13.238.148@KRBTEST.COM.keytab.
I20260929 03:16:15.243125 14266 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskdvlSPq/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/postgres -l logfile start
2026-09-29 03:16:17.782 UTC [25052] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 03:16:17.782 UTC [25052] LOG: listening on IPv4 address "127.13.238.148", port 55293
2026-09-29 03:16:17.786 UTC [25052] LOG: listening on Unix socket "/tmp/.s.PGSQL.55293"
2026-09-29 03:16:17.816 UTC [25055] LOG: database system was shut down at 2026-09-29 03:16:16 UTC
2026-09-29 03:16:17.820 UTC [25052] LOG: database system is ready to accept connections
I20260929 03:16:18.694960 14266 mini_postgres.cc:96] Postgres bound to 55293
2026-09-29 03:16:18.701 UTC [25063] FATAL: database "slave" does not exist
127.13.238.148:55293 - accepting connections
I20260929 03:16:18.702169 14266 mini_ranger.cc:161] Starting Ranger...
I20260929 03:16:18.717156 14266 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 03:16:19.229292 14266 mini_ranger.cc:87] Created ranger Postgres database
I20260929 03:16:19.229370 14266 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ranger-admin
2026-09-29 03:16:19,544 [I] DB FLAVOR :POSTGRES
2026-09-29 03:16:19,545 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 03:16:19,545 [I] Checking connection..
2026-09-29 03:16:19,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:55293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 03:16:19,781 [I] Checking connection passed.
2026-09-29 03:16:19,782 [I] --------- Verifying version history table ---------
2026-09-29 03:16:19,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:55293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:16:20,043 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 03:16:20,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:55293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:16:20,294 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 03:16:20,294 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 03:16:20,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:55293/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 03:16:20.525 UTC [25127] WARNING: there is no transaction in progress
2026-09-29 03:16:20,531 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 03:16:20,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:55293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:16:20,772 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 03:16:20,772 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 03:16:20,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:55293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 03:16:21,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:55293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 03:16:21,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:55293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-nbjg.c.gcp-upstream.internal','N') ;"
2026-09-29 03:16:21,478 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:55293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 03:16:21,706 [I] Table x_portal_user does not exist in database ranger
2026-09-29 03:16:21,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:55293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 03:16:21,935 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 03:16:21,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:55293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 03:16:22,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:55293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 03:16:22,419 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 03:16:22,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:55293/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 03:16:23.398 UTC [25235] WARNING: there is no transaction in progress
2026-09-29 03:16:23.416 UTC [25235] WARNING: there is no transaction in progress
2026-09-29 03:16:23.433 UTC [25235] WARNING: there is no transaction in progress
2026-09-29 03:16:23.449 UTC [25235] WARNING: there is no transaction in progress
2026-09-29 03:16:23.465 UTC [25235] WARNING: there is no transaction in progress
2026-09-29 03:16:23.482 UTC [25235] WARNING: there is no transaction in progress
2026-09-29 03:16:23.541 UTC [25235] WARNING: there is no transaction in progress
2026-09-29 03:16:23.549 UTC [25235] WARNING: there is no transaction in progress
2026-09-29 03:16:23.564 UTC [25235] WARNING: there is no transaction in progress
2026-09-29 03:16:23.574 UTC [25235] WARNING: there is no transaction in progress
2026-09-29 03:16:23.585 UTC [25235] WARNING: there is no transaction in progress
2026-09-29 03:16:23.598 UTC [25235] WARNING: there is no transaction in progress
2026-09-29 03:16:23.608 UTC [25235] WARNING: there is no transaction in progress
2026-09-29 03:16:23.614 UTC [25235] WARNING: there is no transaction in progress
2026-09-29 03:16:23.622 UTC [25235] WARNING: there is no transaction in progress
2026-09-29 03:16:23.629 UTC [25235] WARNING: there is no transaction in progress
2026-09-29 03:16:24,318 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 03:16:24,319 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:55293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 03:16:24,570 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 03:16:24,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:55293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 03:16:24,828 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 03:16:24,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:55293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 03:16:25,096 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 03:16:25,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:55293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 03:16:25,353 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:55293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 03:16:25,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:55293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-nbjg.c.gcp-upstream.internal';"
2026-09-29 03:16:25,819 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 03:16:25,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:55293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 03:16:26,058 [I] DB_PATCHES have already been applied
I20260929 03:16:26.064148 14266 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ranger-admin:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskdvlSPq/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 03:16:26.064215 14266 mini_ranger.cc:193] Using host: 127.13.238.148
I20260929 03:16:26.067602 14266 mini_ranger.cc:227] Ranger admin URL: http://127.13.238.148:60789
Sep 29, 2026 3:16:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:16:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:16:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 3:16:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 3:16:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 3:16:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 3:16:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 03:16:27.064543 14266 mini_ranger.cc:160] Time spent starting Ranger: real 8.362s user 0.000s sys 0.009s
Sep 29, 2026 3:16:47 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 3:16:47 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 3:16:47 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 3:16:47 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 3:16:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 3:16:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 3:16:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 3:16:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 3:16:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 3:16:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 3:16:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 3:16:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 3:16:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 3:16:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 3:16:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 3:16:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 3:16:47 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 3:16:48 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 03:16:49.531828 14266 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.13.238.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.13.238.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.13.238.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/rangerkms_127.13.238.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.13.238.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/rangerkms_127.13.238.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.238.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.13.238.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.238.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/HTTP_127.13.238.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.238.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/HTTP_127.13.238.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 03:16:49 dist-test-slave-nbjg krb5kdc[25022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651809, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 03:16:49.620792 14266 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 03:16:49.630962 14266 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 03:16:50.545523 14266 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 03:16:50.545615 14266 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ranger-kms
I20260929 03:16:50.545709 14266 mini_ranger_kms.cc:227] Using postgres at 127.13.238.148:55293
2026-09-29 03:16:50,756 [I] DB FLAVOR :POSTGRES
2026-09-29 03:16:50,757 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 03:16:50,757 [I] Checking connection
2026-09-29 03:16:50,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:55293/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 03:16:50,989 [I] connection success
2026-09-29 03:16:50,989 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 03:16:50,989 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 03:16:50,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:55293/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 03:16:51,240 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 03:16:51,240 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 03:16:51,240 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 03:16:51,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:55293/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 03:16:51,504 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 03:16:51.976169 14266 mini_ranger_kms.cc:310] Created kms service
I20260929 03:16:52.173817 14266 mini_ranger_kms.cc:326] Created kudu user
I20260929 03:16:52.215725 14266 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 03:16:52 dist-test-slave-nbjg krb5kdc[25022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651809, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.13.238.148@KRBTEST.COM
I20260929 03:16:52.569406 14266 mini_ranger_kms.cc:384] Added ranger policy
I20260929 03:16:52.569581 14266 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ranger-kms:/tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 03:16:52.569603 14266 mini_ranger_kms.cc:240] Using host: 127.13.238.148
I20260929 03:16:52.573694 14266 mini_ranger_kms.cc:276] Ranger KMS PID: 25415
I20260929 03:16:52.573863 14266 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.13.238.148:40225
Sep 29, 2026 3:16:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:16:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:16:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 3:16:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 3:16:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 3:16:54 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 03:16:54.642597 14266 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.022s user 0.006s sys 0.186s
I20260929 03:16:54.642740 14266 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 03:16:54.642866 14266 mini_ranger_kms.cc:401] 127.13.238.148:40225/kms/v1/keys
Sep 29 03:16:54 dist-test-slave-nbjg krb5kdc[25022](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790651814, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.238.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 03:16:55 dist-test-slave-nbjg krb5kdc[25022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651814, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.238.148@KRBTEST.COM for HTTP/127.13.238.148@KRBTEST.COM
Sep 29 03:16:55 dist-test-slave-nbjg krb5kdc[25022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651814, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.238.148@KRBTEST.COM for HTTP/127.13.238.148@KRBTEST.COM
Sep 29 03:16:58 dist-test-slave-nbjg krb5kdc[25022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:16:58.198392 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:45343
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:37071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:45343
--ranger_config_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.238.148:40225/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:16:58.312314 25466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:16:58.312605 25466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:16:58.312664 25466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:16:58.316772 25466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:16:58.316845 25466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:16:58.316871 25466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:16:58.316890 25466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:16:58.316911 25466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:16:58.322523 25466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:37071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/master-0/wal
--ranger_kms_url=127.13.238.148:40225/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:45343
--ranger_config_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:45343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:16:58.323614 25466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:16:58.324635 25466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:16:58.333279 25474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:16:58.333312 25471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:16:58.333422 25472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:16:58.333451 25466 server_base.cc:1034] running on GCE node
I20260929 03:16:58.334139 25466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:16:58.335289 25466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:16:58.336472 25466 hybrid_clock.cc:648] HybridClock initialized: now 1790651818336451 us; error 36 us; skew 500 ppm
Sep 29 03:16:58 dist-test-slave-nbjg krb5kdc[25022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651818, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:16:58.339371 25466 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:16:58.340332 25466 webserver.cc:460] Webserver started at http://127.13.238.190:44129/ using document root <none> and password file <none>
I20260929 03:16:58.340873 25466 fs_manager.cc:359] Metadata directory not provided
I20260929 03:16:58.340941 25466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:16:58.341140 25466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:16:58 dist-test-slave-nbjg krb5kdc[25022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651818, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for HTTP/127.13.238.148@KRBTEST.COM
I20260929 03:16:58.402411 25466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "7903e7509366430e9082f9118a8bf932"
format_stamp: "Formatted at 2026-09-29 03:16:58 on dist-test-slave-nbjg"
server_key: "4fa539221cd0bc7439f15d5b73ba636b"
server_key_iv: "39f56c7b8634a6de7763a7181971eb26"
server_key_version: "kuduclusterkey@0"
I20260929 03:16:58.403144 25466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "7903e7509366430e9082f9118a8bf932"
format_stamp: "Formatted at 2026-09-29 03:16:58 on dist-test-slave-nbjg"
server_key: "4fa539221cd0bc7439f15d5b73ba636b"
server_key_iv: "39f56c7b8634a6de7763a7181971eb26"
server_key_version: "kuduclusterkey@0"
I20260929 03:16:58.408073 25466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:16:58.424347 25481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:58.425921 25466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:16:58.426141 25466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "7903e7509366430e9082f9118a8bf932"
format_stamp: "Formatted at 2026-09-29 03:16:58 on dist-test-slave-nbjg"
server_key: "4fa539221cd0bc7439f15d5b73ba636b"
server_key_iv: "39f56c7b8634a6de7763a7181971eb26"
server_key_version: "kuduclusterkey@0"
I20260929 03:16:58.426371 25466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:16:58.461043 25466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:16:58.464330 25466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:16:58.464516 25466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:16:58.472923 25466 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:45343
I20260929 03:16:58.472946 25533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:45343 every 8 connection(s)
I20260929 03:16:58.474177 25466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:16:58.476527 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 25466
I20260929 03:16:58.476650 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:16:58.476946 14266 external_mini_cluster.cc:1428] Setting key 658f130836fa965e13db777159904941
I20260929 03:16:58.477391 25534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:16:58.485677 25534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7903e7509366430e9082f9118a8bf932: Bootstrap starting.
Sep 29 03:16:58 dist-test-slave-nbjg krb5kdc[25022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:16:58.488200 25534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7903e7509366430e9082f9118a8bf932: Neither blocks nor log segments found. Creating new log.
I20260929 03:16:58.489377 25534 log.cc:824] T 00000000000000000000000000000000 P 7903e7509366430e9082f9118a8bf932: Log is configured to *not* fsync() on all Append() calls
I20260929 03:16:58.491511 25534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7903e7509366430e9082f9118a8bf932: No bootstrap required, opened a new log
I20260929 03:16:58.493651 25537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:58.478434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43560 (local address 127.13.238.190:45343)
0929 03:16:58.478926 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:58.478944 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:58.478987 (+ 43us) server_negotiation.cc:407] Connection header received
0929 03:16:58.479869 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:58.479894 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:58.480339 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:58.480707 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:58.481643 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:58.483177 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:58.484410 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:58.485091 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:58.485251 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:58.488171 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:16:58.488178 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:16:58.488195 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:16:58.488226 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:16:58.490065 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:16:58.490579 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:16:58.490585 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:16:58.490591 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:16:58.490719 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:16:58.491016 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:16:58.491019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:16:58.491021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:16:58.491514 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:16:58.492246 (+ 732us) server_negotiation.cc:1035] Waiting for connection context
0929 03:16:58.492406 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 03:16:58.492697 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":148,"threads_started":1}
I20260929 03:16:58.494843 25534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7903e7509366430e9082f9118a8bf932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7903e7509366430e9082f9118a8bf932" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 45343 } }
I20260929 03:16:58.495134 25534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7903e7509366430e9082f9118a8bf932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:16:58.495227 25534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7903e7509366430e9082f9118a8bf932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7903e7509366430e9082f9118a8bf932, State: Initialized, Role: FOLLOWER
I20260929 03:16:58.495816 25534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7903e7509366430e9082f9118a8bf932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7903e7509366430e9082f9118a8bf932" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 45343 } }
I20260929 03:16:58.496097 25534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7903e7509366430e9082f9118a8bf932 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:16:58.496222 25534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7903e7509366430e9082f9118a8bf932 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:16:58.496336 25534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7903e7509366430e9082f9118a8bf932 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:16:58.497225 25534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7903e7509366430e9082f9118a8bf932 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7903e7509366430e9082f9118a8bf932" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 45343 } }
I20260929 03:16:58.497570 25534 leader_election.cc:302] T 00000000000000000000000000000000 P 7903e7509366430e9082f9118a8bf932 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7903e7509366430e9082f9118a8bf932; no voters:
I20260929 03:16:58.497864 25534 leader_election.cc:288] T 00000000000000000000000000000000 P 7903e7509366430e9082f9118a8bf932 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:16:58.497967 25539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7903e7509366430e9082f9118a8bf932 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:16:58.498215 25539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7903e7509366430e9082f9118a8bf932 [term 1 LEADER]: Becoming Leader. State: Replica: 7903e7509366430e9082f9118a8bf932, State: Running, Role: LEADER
I20260929 03:16:58.498593 25539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7903e7509366430e9082f9118a8bf932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7903e7509366430e9082f9118a8bf932" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 45343 } }
I20260929 03:16:58.499480 25534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7903e7509366430e9082f9118a8bf932 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:16:58.499900 25539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7903e7509366430e9082f9118a8bf932 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7903e7509366430e9082f9118a8bf932" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7903e7509366430e9082f9118a8bf932" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 45343 } } }
I20260929 03:16:58.500093 25539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7903e7509366430e9082f9118a8bf932 [sys.catalog]: This master's current role is: LEADER
I20260929 03:16:58.500349 25539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7903e7509366430e9082f9118a8bf932 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7903e7509366430e9082f9118a8bf932. Latest consensus state: current_term: 1 leader_uuid: "7903e7509366430e9082f9118a8bf932" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7903e7509366430e9082f9118a8bf932" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 45343 } } }
I20260929 03:16:58.500590 25539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7903e7509366430e9082f9118a8bf932 [sys.catalog]: This master's current role is: LEADER
I20260929 03:16:58.500926 25543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:16:58.504521 25543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:16:58.504940 25534 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 03:16:58.517755 25543 catalog_manager.cc:1347] Generated new cluster ID: b890b21c5764479584cd9ebf03810972
I20260929 03:16:58.517849 25543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:16:58.540719 25543 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:16:58.542313 25543 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:16:58.558858 25543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7903e7509366430e9082f9118a8bf932: Generated new TSK 0
I20260929 03:16:58.559934 25543 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
I20260929 03:16:59.293157 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.238.148:40225/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:0
--local_ip_for_outbound_sockets=127.13.238.129
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:45343
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:37071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:16:59.409029 25579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:16:59.409283 25579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:16:59.409340 25579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:16:59.413455 25579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:16:59.413509 25579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:16:59.413584 25579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:16:59.419140 25579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:37071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.13.238.148:40225/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:45343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:16:59.420260 25579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:16:59.421322 25579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:16:59.431799 25584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:16:59.431960 25587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:16:59.431799 25585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:16:59.432328 25579 server_base.cc:1034] running on GCE node
I20260929 03:16:59.432900 25579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:16:59.433688 25579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:16:59.434875 25579 hybrid_clock.cc:648] HybridClock initialized: now 1790651819434796 us; error 95 us; skew 500 ppm
Sep 29 03:16:59 dist-test-slave-nbjg krb5kdc[25022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651819, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:16:59.437798 25579 init.cc:377] Logged in from keytab as kudu/127.13.238.129@KRBTEST.COM (short username kudu)
I20260929 03:16:59.438733 25579 webserver.cc:460] Webserver started at http://127.13.238.129:37791/ using document root <none> and password file <none>
I20260929 03:16:59.439424 25579 fs_manager.cc:359] Metadata directory not provided
I20260929 03:16:59.439522 25579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:16:59.439749 25579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:16:59 dist-test-slave-nbjg krb5kdc[25022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651819, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for HTTP/127.13.238.148@KRBTEST.COM
I20260929 03:16:59.456321 25579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-0/data/instance:
uuid: "7f31e16136494e59936a5cccfb4813d8"
format_stamp: "Formatted at 2026-09-29 03:16:59 on dist-test-slave-nbjg"
server_key: "afd9667fd19f6876d0d6521cf9f05629"
server_key_iv: "584fc5fc7cd43229f65f33135cf9a666"
server_key_version: "kuduclusterkey@0"
I20260929 03:16:59.457093 25579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance:
uuid: "7f31e16136494e59936a5cccfb4813d8"
format_stamp: "Formatted at 2026-09-29 03:16:59 on dist-test-slave-nbjg"
server_key: "afd9667fd19f6876d0d6521cf9f05629"
server_key_iv: "584fc5fc7cd43229f65f33135cf9a666"
server_key_version: "kuduclusterkey@0"
I20260929 03:16:59.462320 25579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 03:16:59.490888 25596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:59.492794 25579 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.001s
I20260929 03:16:59.492949 25579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "7f31e16136494e59936a5cccfb4813d8"
format_stamp: "Formatted at 2026-09-29 03:16:59 on dist-test-slave-nbjg"
server_key: "afd9667fd19f6876d0d6521cf9f05629"
server_key_iv: "584fc5fc7cd43229f65f33135cf9a666"
server_key_version: "kuduclusterkey@0"
I20260929 03:16:59.493091 25579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:16:59.513589 25579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:16:59.517359 25579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:16:59.517642 25579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:16:59.518429 25579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:16:59.519593 25579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:16:59.519702 25579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:59.519827 25579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:16:59.519899 25579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:59.533433 25579 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:45617
I20260929 03:16:59.533664 25710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:45617 every 8 connection(s)
I20260929 03:16:59.534713 25579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:16:59.541626 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 25579
I20260929 03:16:59.541752 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance
I20260929 03:16:59.542035 14266 external_mini_cluster.cc:1428] Setting key 85f34c55fbb5425cfafc7836d3da7c03
WARNING: no policy specified for kudu/127.13.238.130@KRBTEST.COM; defaulting to no policy
Sep 29 03:16:59 dist-test-slave-nbjg krb5kdc[25022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651819, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.130@KRBTEST.COM" created.
I20260929 03:16:59.578826 25714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:59.537773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:45343 (local address 127.13.238.129:45397)
0929 03:16:59.550608 (+ 12835us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:59.550693 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:59.551824 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:59.552907 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:59.552919 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:59.553325 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:59.554067 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:59.554088 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:59.556663 (+ 2575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:59.556667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:59.558148 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:59.558160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:59.570403 (+ 12243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:59.570410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:59.570714 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:59.571489 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:59.571513 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:59.573176 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:16:59.575325 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:16:59.575333 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:16:59.575355 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:16:59.575717 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:16:59.576051 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:16:59.576055 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:16:59.576057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:16:59.576255 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:16:59.576726 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:16:59.576733 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:16:59.577078 (+ 345us) client_negotiation.cc:769] Sending connection context
0929 03:16:59.577312 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 03:16:59.577685 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12522,"thread_start_us":132,"threads_started":1}
I20260929 03:16:59.579584 25715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:59.538904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:45397 (local address 127.13.238.190:45343)
0929 03:16:59.552341 (+ 13437us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:59.552351 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:59.552367 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:16:59.552410 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:59.552413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:59.552462 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:59.552548 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:59.554267 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:59.555135 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:59.558323 (+ 3188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:59.558685 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:59.558783 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:59.573425 (+ 14642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:16:59.573430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:16:59.573433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:16:59.573465 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:16:59.575186 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:16:59.575838 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:16:59.575842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:16:59.575843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:16:59.575943 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:16:59.576370 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:16:59.576373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:16:59.576375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:16:59.576598 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:16:59.576675 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:16:59.579295 (+ 2620us) server_negotiation.cc:299] Negotiation successful
0929 03:16:59.579429 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13337,"thread_start_us":83,"threads_started":1}
I20260929 03:16:59.581483 25712 heartbeater.cc:347] Connected to a master server at 127.13.238.190:45343
I20260929 03:16:59.581746 25712 heartbeater.cc:464] Registering TS with master...
I20260929 03:16:59.582402 25712 heartbeater.cc:511] Master 127.13.238.190:45343 requested a full tablet report, sending...
I20260929 03:16:59.584355 25498 ts_manager.cc:194] Registered new tserver with Master: 7f31e16136494e59936a5cccfb4813d8 (127.13.238.129:45617)
I20260929 03:16:59.585657 25498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.129@KRBTEST.COM'} at 127.13.238.129:45397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
I20260929 03:16:59.591499 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.238.148:40225/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:0
--local_ip_for_outbound_sockets=127.13.238.130
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:45343
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:37071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:16:59 dist-test-slave-nbjg krb5kdc[25022](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790651819, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 03:16:59.718432 25718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:16:59.718786 25718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:16:59.718879 25718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:16:59.723837 25718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:16:59.723902 25718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:16:59.723982 25718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:16:59.729969 25718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:37071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.13.238.148:40225/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:45343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:16:59.731150 25718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:16:59.732576 25718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:16:59.744830 25725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:16:59.744894 25724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:16:59.744868 25727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:16:59.745247 25718 server_base.cc:1034] running on GCE node
I20260929 03:16:59.745774 25718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:16:59.746492 25718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:16:59.747697 25718 hybrid_clock.cc:648] HybridClock initialized: now 1790651819747662 us; error 42 us; skew 500 ppm
Sep 29 03:16:59 dist-test-slave-nbjg krb5kdc[25022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651819, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:16:59.750670 25718 init.cc:377] Logged in from keytab as kudu/127.13.238.130@KRBTEST.COM (short username kudu)
I20260929 03:16:59.751669 25718 webserver.cc:460] Webserver started at http://127.13.238.130:36859/ using document root <none> and password file <none>
I20260929 03:16:59.752233 25718 fs_manager.cc:359] Metadata directory not provided
I20260929 03:16:59.752291 25718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:16:59.752509 25718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:16:59 dist-test-slave-nbjg krb5kdc[25022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651819, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for HTTP/127.13.238.148@KRBTEST.COM
I20260929 03:16:59.768354 25718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-1/data/instance:
uuid: "5ffbcb61799d4a51adea1f502f994416"
format_stamp: "Formatted at 2026-09-29 03:16:59 on dist-test-slave-nbjg"
server_key: "431be9823a0f43e56ed652095c0b2d08"
server_key_iv: "31e9219e34277e12c5b14450299a7f19"
server_key_version: "kuduclusterkey@0"
I20260929 03:16:59.769124 25718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance:
uuid: "5ffbcb61799d4a51adea1f502f994416"
format_stamp: "Formatted at 2026-09-29 03:16:59 on dist-test-slave-nbjg"
server_key: "431be9823a0f43e56ed652095c0b2d08"
server_key_iv: "31e9219e34277e12c5b14450299a7f19"
server_key_version: "kuduclusterkey@0"
I20260929 03:16:59.774719 25718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 03:16:59.802707 25734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:59.804587 25718 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260929 03:16:59.804812 25718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "5ffbcb61799d4a51adea1f502f994416"
format_stamp: "Formatted at 2026-09-29 03:16:59 on dist-test-slave-nbjg"
server_key: "431be9823a0f43e56ed652095c0b2d08"
server_key_iv: "31e9219e34277e12c5b14450299a7f19"
server_key_version: "kuduclusterkey@0"
I20260929 03:16:59.805045 25718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:16:59.827029 25718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:16:59.830950 25718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:16:59.831189 25718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:16:59.831923 25718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:16:59.832921 25718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:16:59.832973 25718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:59.833027 25718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:16:59.833055 25718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:16:59.843892 25718 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:35463
I20260929 03:16:59.845206 25718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:16:59.846895 25847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:35463 every 8 connection(s)
I20260929 03:16:59.850436 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 25718
I20260929 03:16:59.850559 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance
I20260929 03:16:59.850884 14266 external_mini_cluster.cc:1428] Setting key 6931c3a8102569cf44fc782376210722
Sep 29 03:16:59 dist-test-slave-nbjg krb5kdc[25022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651819, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:16:59.864735 25715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:59.850873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:34277 (local address 127.13.238.190:45343)
0929 03:16:59.851009 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 03:16:59.851014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:16:59.851813 (+ 799us) server_negotiation.cc:407] Connection header received
0929 03:16:59.853040 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:16:59.853044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:16:59.853095 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:16:59.853150 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:16:59.854825 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:59.855676 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:59.857370 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:16:59.857726 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:16:59.857805 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:59.860811 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:16:59.860816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:16:59.860818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:16:59.860847 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:16:59.862397 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:16:59.863015 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:16:59.863019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:16:59.863020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:16:59.863099 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:16:59.863451 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:16:59.863454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:16:59.863456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:16:59.863656 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:16:59.863726 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:16:59.864431 (+ 705us) server_negotiation.cc:299] Negotiation successful
0929 03:16:59.864563 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:16:59.866173 25850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:16:59.849692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:45343 (local address 127.13.238.130:34277)
0929 03:16:59.851651 (+ 1959us) negotiation.cc:107] Waiting for socket to connect
0929 03:16:59.851711 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 03:16:59.852787 (+ 1076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:16:59.853578 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:16:59.853587 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:16:59.853937 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:16:59.854646 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:59.854668 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:59.855836 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:59.855839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:59.857196 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:16:59.857207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:16:59.858009 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:16:59.858013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:16:59.858236 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:16:59.858914 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:16:59.858936 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:16:59.860623 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:16:59.862503 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:16:59.862510 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:16:59.862528 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:16:59.862906 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:16:59.863190 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:16:59.863193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:16:59.863194 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:16:59.863358 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:16:59.863789 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:16:59.863795 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:16:59.864139 (+ 344us) client_negotiation.cc:769] Sending connection context
0929 03:16:59.865108 (+ 969us) client_negotiation.cc:240] Negotiation successful
0929 03:16:59.865381 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1728,"thread_start_us":147,"threads_started":1}
I20260929 03:16:59.867695 25848 heartbeater.cc:347] Connected to a master server at 127.13.238.190:45343
I20260929 03:16:59.867950 25848 heartbeater.cc:464] Registering TS with master...
I20260929 03:16:59.868553 25848 heartbeater.cc:511] Master 127.13.238.190:45343 requested a full tablet report, sending...
I20260929 03:16:59.869783 25498 ts_manager.cc:194] Registered new tserver with Master: 5ffbcb61799d4a51adea1f502f994416 (127.13.238.130:35463)
WARNING: no policy specified for kudu/127.13.238.131@KRBTEST.COM; defaulting to no policy
I20260929 03:16:59.870569 25498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.130@KRBTEST.COM'} at 127.13.238.130:34277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
I20260929 03:16:59.899830 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.238.148:40225/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:0
--local_ip_for_outbound_sockets=127.13.238.131
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:45343
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:37071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:00.060704 25853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:00.061214 25853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:00.061363 25853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:00.067431 25853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:00.067637 25853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:00.067781 25853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:17:00.075985 25853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:37071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.13.238.148:40225/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:45343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:17:00.077766 25853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:00.079319 25853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:00.093271 25858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:00.093808 25859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:00.094399 25853 server_base.cc:1034] running on GCE node
W20260929 03:17:00.094221 25861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:00.100219 25853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:00.101393 25853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:00.108891 25853 hybrid_clock.cc:648] HybridClock initialized: now 1790651820108780 us; error 117 us; skew 500 ppm
Sep 29 03:17:00 dist-test-slave-nbjg krb5kdc[25022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651819, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for HTTP/127.13.238.148@KRBTEST.COM
Sep 29 03:17:00 dist-test-slave-nbjg krb5kdc[25022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651820, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:00.119068 25853 init.cc:377] Logged in from keytab as kudu/127.13.238.131@KRBTEST.COM (short username kudu)
I20260929 03:17:00.120735 25853 webserver.cc:460] Webserver started at http://127.13.238.131:35463/ using document root <none> and password file <none>
I20260929 03:17:00.121665 25853 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:00.121858 25853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:00.122185 25853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:17:00 dist-test-slave-nbjg krb5kdc[25022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651820, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for HTTP/127.13.238.148@KRBTEST.COM
I20260929 03:17:00.143646 25853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-2/data/instance:
uuid: "105e06477df0449395f88e408dee85e7"
format_stamp: "Formatted at 2026-09-29 03:17:00 on dist-test-slave-nbjg"
server_key: "e42ab078d53e1cae3f2c01e8c9edc52f"
server_key_iv: "e6c62c33f26545638f3f9f14272046ee"
server_key_version: "kuduclusterkey@0"
I20260929 03:17:00.144347 25853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance:
uuid: "105e06477df0449395f88e408dee85e7"
format_stamp: "Formatted at 2026-09-29 03:17:00 on dist-test-slave-nbjg"
server_key: "e42ab078d53e1cae3f2c01e8c9edc52f"
server_key_iv: "e6c62c33f26545638f3f9f14272046ee"
server_key_version: "kuduclusterkey@0"
I20260929 03:17:00.151531 25853 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260929 03:17:00.168596 25868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:00.170408 25853 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 03:17:00.170548 25853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "105e06477df0449395f88e408dee85e7"
format_stamp: "Formatted at 2026-09-29 03:17:00 on dist-test-slave-nbjg"
server_key: "e42ab078d53e1cae3f2c01e8c9edc52f"
server_key_iv: "e6c62c33f26545638f3f9f14272046ee"
server_key_version: "kuduclusterkey@0"
I20260929 03:17:00.170687 25853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:00.239585 25853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:00.245129 25853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:00.245397 25853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:00.246260 25853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:00.247365 25853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:00.247488 25853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:00.247612 25853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:00.247679 25853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:00.258039 25853 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:44413
I20260929 03:17:00.259374 25853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:17:00.267740 25981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:44413 every 8 connection(s)
I20260929 03:17:00.274554 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 25853
I20260929 03:17:00.274691 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance
I20260929 03:17:00.275040 14266 external_mini_cluster.cc:1428] Setting key ce009a52ff14368415062bc2e3c7ef05
Sep 29 03:17:00 dist-test-slave-nbjg krb5kdc[25022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651820, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:17:00.297538 25715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:00.276800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:56221 (local address 127.13.238.190:45343)
0929 03:17:00.277069 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:00.277074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:00.283369 (+ 6295us) server_negotiation.cc:407] Connection header received
0929 03:17:00.284631 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:00.284635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:00.284694 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:00.284765 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:00.286599 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.287445 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.289288 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.289636 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.289742 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:00.293386 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:00.293390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:00.293393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:00.293423 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:00.295174 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:00.295773 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:00.295777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:00.295778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:00.295879 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:00.296263 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:00.296266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:00.296268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:00.296482 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:00.296555 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:00.297222 (+ 667us) server_negotiation.cc:299] Negotiation successful
0929 03:17:00.297377 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260929 03:17:00.299445 25984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:00.277409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:45343 (local address 127.13.238.131:56221)
0929 03:17:00.283162 (+ 5753us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:00.283249 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:00.284381 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:00.285137 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:00.285148 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:00.285558 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:00.286396 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:00.286422 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.287630 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.287633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:00.289135 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:00.289149 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.290337 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.290342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:00.290551 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:00.291482 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:00.291506 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:00.293189 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:00.295299 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:00.295306 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:00.295326 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:00.295664 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:00.295979 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:00.295981 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:00.295983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:00.296163 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:00.296617 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:00.296624 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:00.296959 (+ 335us) client_negotiation.cc:769] Sending connection context
0929 03:17:00.298007 (+ 1048us) client_negotiation.cc:240] Negotiation successful
0929 03:17:00.298388 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5441,"thread_start_us":151,"threads_started":1}
I20260929 03:17:00.300940 25982 heartbeater.cc:347] Connected to a master server at 127.13.238.190:45343
I20260929 03:17:00.301179 25982 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:00.301728 25982 heartbeater.cc:511] Master 127.13.238.190:45343 requested a full tablet report, sending...
I20260929 03:17:00.303145 25497 ts_manager.cc:194] Registered new tserver with Master: 105e06477df0449395f88e408dee85e7 (127.13.238.131:44413)
I20260929 03:17:00.303731 25497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.131@KRBTEST.COM'} at 127.13.238.131:56221
I20260929 03:17:00.305013 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:17:00.333796 25715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:00.314842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43562 (local address 127.13.238.190:45343)
0929 03:17:00.314982 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:00.314987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:00.324939 (+ 9952us) server_negotiation.cc:407] Connection header received
0929 03:17:00.325079 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:00.325082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:00.325145 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:00.325250 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:00.326109 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.326919 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.328103 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.328458 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.328563 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:00.330186 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:00.330190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:00.330219 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:00.330246 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:00.331907 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:00.332467 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:00.332470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:00.332473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:00.332557 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:00.332841 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:00.332845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:00.332847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:00.333062 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:00.333159 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:00.333512 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 03:17:00.333629 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 03:17:00.337775 25498 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:17:00.342127 25498 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:17:00.363803 25994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:00.352172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:44413 (local address 127.0.0.1:50468)
0929 03:17:00.356134 (+ 3962us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:00.356190 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:00.356346 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:00.357343 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:00.357349 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:00.357373 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:00.357508 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:00.357515 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.359230 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.359233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:00.360644 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:00.360655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.363252 (+ 2597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.363256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:00.363379 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:00.363421 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 03:17:00.363524 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 03:17:00.363644 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3774,"thread_start_us":98,"threads_started":1}
I20260929 03:17:00.364643 25995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:00.353052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50468 (local address 127.13.238.131:44413)
0929 03:17:00.356663 (+ 3611us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:00.356673 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:00.356698 (+ 25us) server_negotiation.cc:407] Connection header received
0929 03:17:00.356773 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:00.356779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:00.357012 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:00.357151 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:00.357895 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.359104 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.362059 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.362944 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.363060 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:00.363193 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:00.364398 (+ 1205us) server_negotiation.cc:299] Negotiation successful
0929 03:17:00.364503 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3499,"thread_start_us":102,"threads_started":1}
I20260929 03:17:00.368059 25898 tablet_service.cc:1467] Processing CreateTablet for tablet 417c9f14cc2944a5a58127fb94a6e373 (DEFAULT_TABLE table=test-table [id=8d9c8898c6b74cd88fa8038e51ae97ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:00.369230 25898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 417c9f14cc2944a5a58127fb94a6e373. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:00.381783 25997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:00.354412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52736 (local address 127.13.238.129:45617)
0929 03:17:00.361181 (+ 6769us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:00.361188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:00.361212 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:17:00.361286 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:00.361292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:00.361505 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:00.361643 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:00.366891 (+ 5248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.368028 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.380021 (+ 11993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.380878 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.380996 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:00.381141 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:00.381512 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 03:17:00.381625 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6656,"thread_start_us":108,"threads_started":1}
I20260929 03:17:00.382438 25996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:00.354003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:45617 (local address 127.0.0.1:52736)
0929 03:17:00.357702 (+ 3699us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:00.357714 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:00.357785 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:00.361791 (+ 4006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:00.361794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:00.361812 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:00.361923 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:00.361930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.375745 (+ 13815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.375749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:00.377304 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:00.377313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.381238 (+ 3925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.381241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:00.381349 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:00.381372 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:17:00.382219 (+ 847us) client_negotiation.cc:240] Negotiation successful
0929 03:17:00.382304 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3639,"thread_start_us":85,"threads_started":1}
I20260929 03:17:00.385898 25644 tablet_service.cc:1467] Processing CreateTablet for tablet 417c9f14cc2944a5a58127fb94a6e373 (DEFAULT_TABLE table=test-table [id=8d9c8898c6b74cd88fa8038e51ae97ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:00.387109 25644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 417c9f14cc2944a5a58127fb94a6e373. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:00.396391 25999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:00.355492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33480 (local address 127.13.238.130:35463)
0929 03:17:00.368307 (+ 12815us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:00.368322 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:00.368344 (+ 22us) server_negotiation.cc:407] Connection header received
0929 03:17:00.368430 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:00.368435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:00.368632 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:00.368779 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:00.377452 (+ 8673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.378590 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.394444 (+ 15854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.395340 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.395465 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:00.395602 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:00.396114 (+ 512us) server_negotiation.cc:299] Negotiation successful
0929 03:17:00.396233 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12699,"thread_start_us":66,"threads_started":1}
I20260929 03:17:00.397050 25998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:00.355118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:35463 (local address 127.0.0.1:33480)
0929 03:17:00.360856 (+ 5738us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:00.360868 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:00.360946 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:00.376869 (+ 15923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:00.376873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:00.376905 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:00.376999 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:00.377005 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.378721 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.378724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:00.379853 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:00.379864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.395724 (+ 15860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.395728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:00.395831 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:00.395856 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 03:17:00.396786 (+ 930us) client_negotiation.cc:240] Negotiation successful
0929 03:17:00.396914 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5678,"thread_start_us":87,"threads_started":1}
I20260929 03:17:00.400883 25770 tablet_service.cc:1467] Processing CreateTablet for tablet 417c9f14cc2944a5a58127fb94a6e373 (DEFAULT_TABLE table=test-table [id=8d9c8898c6b74cd88fa8038e51ae97ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:00.402280 25770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 417c9f14cc2944a5a58127fb94a6e373. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:00.407094 26001 tablet_bootstrap.cc:492] T 417c9f14cc2944a5a58127fb94a6e373 P 7f31e16136494e59936a5cccfb4813d8: Bootstrap starting.
I20260929 03:17:00.412333 26002 tablet_bootstrap.cc:492] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416: Bootstrap starting.
I20260929 03:17:00.414456 26002 tablet_bootstrap.cc:654] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:00.414961 26000 tablet_bootstrap.cc:492] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7: Bootstrap starting.
I20260929 03:17:00.415452 26002 log.cc:824] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:00.417778 26000 tablet_bootstrap.cc:654] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:00.417800 26002 tablet_bootstrap.cc:492] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416: No bootstrap required, opened a new log
I20260929 03:17:00.414374 26001 tablet_bootstrap.cc:654] T 417c9f14cc2944a5a58127fb94a6e373 P 7f31e16136494e59936a5cccfb4813d8: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:00.417999 26002 ts_tablet_manager.cc:1397] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 03:17:00.418838 26001 log.cc:824] T 417c9f14cc2944a5a58127fb94a6e373 P 7f31e16136494e59936a5cccfb4813d8: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:00.420009 26000 log.cc:824] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:00.422797 26002 raft_consensus.cc:348] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105e06477df0449395f88e408dee85e7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 44413 } } peers { permanent_uuid: "7f31e16136494e59936a5cccfb4813d8" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 45617 } } peers { permanent_uuid: "5ffbcb61799d4a51adea1f502f994416" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35463 } }
I20260929 03:17:00.423213 26002 raft_consensus.cc:374] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:00.423316 26002 raft_consensus.cc:729] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ffbcb61799d4a51adea1f502f994416, State: Initialized, Role: FOLLOWER
I20260929 03:17:00.423936 26002 consensus_queue.cc:260] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105e06477df0449395f88e408dee85e7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 44413 } } peers { permanent_uuid: "7f31e16136494e59936a5cccfb4813d8" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 45617 } } peers { permanent_uuid: "5ffbcb61799d4a51adea1f502f994416" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35463 } }
I20260929 03:17:00.425155 26002 ts_tablet_manager.cc:1428] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 03:17:00.425580 25848 heartbeater.cc:503] Master 127.13.238.190:45343 was elected leader, sending a full tablet report...
I20260929 03:17:00.432781 26001 tablet_bootstrap.cc:492] T 417c9f14cc2944a5a58127fb94a6e373 P 7f31e16136494e59936a5cccfb4813d8: No bootstrap required, opened a new log
I20260929 03:17:00.432997 26001 ts_tablet_manager.cc:1397] T 417c9f14cc2944a5a58127fb94a6e373 P 7f31e16136494e59936a5cccfb4813d8: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.004s
I20260929 03:17:00.435678 26000 tablet_bootstrap.cc:492] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7: No bootstrap required, opened a new log
I20260929 03:17:00.435894 26000 ts_tablet_manager.cc:1397] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20260929 03:17:00.437098 26001 raft_consensus.cc:348] T 417c9f14cc2944a5a58127fb94a6e373 P 7f31e16136494e59936a5cccfb4813d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105e06477df0449395f88e408dee85e7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 44413 } } peers { permanent_uuid: "7f31e16136494e59936a5cccfb4813d8" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 45617 } } peers { permanent_uuid: "5ffbcb61799d4a51adea1f502f994416" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35463 } }
I20260929 03:17:00.437398 26001 raft_consensus.cc:374] T 417c9f14cc2944a5a58127fb94a6e373 P 7f31e16136494e59936a5cccfb4813d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:00.437445 26001 raft_consensus.cc:729] T 417c9f14cc2944a5a58127fb94a6e373 P 7f31e16136494e59936a5cccfb4813d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f31e16136494e59936a5cccfb4813d8, State: Initialized, Role: FOLLOWER
I20260929 03:17:00.437963 26001 consensus_queue.cc:260] T 417c9f14cc2944a5a58127fb94a6e373 P 7f31e16136494e59936a5cccfb4813d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105e06477df0449395f88e408dee85e7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 44413 } } peers { permanent_uuid: "7f31e16136494e59936a5cccfb4813d8" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 45617 } } peers { permanent_uuid: "5ffbcb61799d4a51adea1f502f994416" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35463 } }
I20260929 03:17:00.439057 26001 ts_tablet_manager.cc:1428] T 417c9f14cc2944a5a58127fb94a6e373 P 7f31e16136494e59936a5cccfb4813d8: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260929 03:17:00.439478 25712 heartbeater.cc:503] Master 127.13.238.190:45343 was elected leader, sending a full tablet report...
I20260929 03:17:00.439888 26000 raft_consensus.cc:348] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105e06477df0449395f88e408dee85e7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 44413 } } peers { permanent_uuid: "7f31e16136494e59936a5cccfb4813d8" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 45617 } } peers { permanent_uuid: "5ffbcb61799d4a51adea1f502f994416" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35463 } }
I20260929 03:17:00.440129 26000 raft_consensus.cc:374] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:00.440176 26000 raft_consensus.cc:729] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 105e06477df0449395f88e408dee85e7, State: Initialized, Role: FOLLOWER
I20260929 03:17:00.440675 26000 consensus_queue.cc:260] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105e06477df0449395f88e408dee85e7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 44413 } } peers { permanent_uuid: "7f31e16136494e59936a5cccfb4813d8" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 45617 } } peers { permanent_uuid: "5ffbcb61799d4a51adea1f502f994416" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35463 } }
I20260929 03:17:00.441747 26000 ts_tablet_manager.cc:1428] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 03:17:00.442094 25982 heartbeater.cc:503] Master 127.13.238.190:45343 was elected leader, sending a full tablet report...
I20260929 03:17:00.461177 26006 raft_consensus.cc:482] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:17:00.461583 26006 raft_consensus.cc:504] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105e06477df0449395f88e408dee85e7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 44413 } } peers { permanent_uuid: "7f31e16136494e59936a5cccfb4813d8" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 45617 } } peers { permanent_uuid: "5ffbcb61799d4a51adea1f502f994416" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35463 } }
I20260929 03:17:00.465145 26006 leader_election.cc:288] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 105e06477df0449395f88e408dee85e7 (127.13.238.131:44413), 7f31e16136494e59936a5cccfb4813d8 (127.13.238.129:45617)
I20260929 03:17:00.465448 26008 raft_consensus.cc:482] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:17:00.465729 26008 raft_consensus.cc:504] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105e06477df0449395f88e408dee85e7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 44413 } } peers { permanent_uuid: "7f31e16136494e59936a5cccfb4813d8" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 45617 } } peers { permanent_uuid: "5ffbcb61799d4a51adea1f502f994416" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35463 } }
I20260929 03:17:00.469941 26008 leader_election.cc:288] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f31e16136494e59936a5cccfb4813d8 (127.13.238.129:45617), 5ffbcb61799d4a51adea1f502f994416 (127.13.238.130:35463)
I20260929 03:17:00.484598 25995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:00.464109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:60825 (local address 127.13.238.131:44413)
0929 03:17:00.464235 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:00.464241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:00.470418 (+ 6177us) server_negotiation.cc:407] Connection header received
0929 03:17:00.470630 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:00.470633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:00.470684 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:00.470732 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:00.471418 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.472162 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.480833 (+ 8671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.481413 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.481514 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:00.481567 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:00.484368 (+ 2801us) server_negotiation.cc:299] Negotiation successful
0929 03:17:00.484457 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:17:00.485291 26010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:00.463653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:44413 (local address 127.13.238.130:60825)
0929 03:17:00.470329 (+ 6676us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:00.470345 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:00.470501 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:00.470904 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:00.470907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:00.470942 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:00.471066 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:00.471072 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.472337 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.472340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:00.480227 (+ 7887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:00.480238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.481674 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.481677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:00.481776 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:00.481799 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:17:00.485031 (+ 3232us) client_negotiation.cc:240] Negotiation successful
0929 03:17:00.485123 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6599,"thread_start_us":164,"threads_started":1}
I20260929 03:17:00.485786 25997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:00.464982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:39335 (local address 127.13.238.129:45617)
0929 03:17:00.465106 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:00.465111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:00.471344 (+ 6233us) server_negotiation.cc:407] Connection header received
0929 03:17:00.473077 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:00.473081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:00.473132 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:00.473175 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:00.473825 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.474587 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.483221 (+ 8634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.483829 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.483922 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:00.483974 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:00.485584 (+ 1610us) server_negotiation.cc:299] Negotiation successful
0929 03:17:00.485671 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 03:17:00.487087 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417c9f14cc2944a5a58127fb94a6e373" candidate_uuid: "5ffbcb61799d4a51adea1f502f994416" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "105e06477df0449395f88e408dee85e7" is_pre_election: true
I20260929 03:17:00.487358 25925 raft_consensus.cc:2454] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ffbcb61799d4a51adea1f502f994416 in term 0.
I20260929 03:17:00.486436 26009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:00.464581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:45617 (local address 127.13.238.130:39335)
0929 03:17:00.471266 (+ 6685us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:00.471279 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:00.472931 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:00.473302 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:00.473305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:00.473321 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:00.473439 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:00.473447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.474728 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.474731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:00.480676 (+ 5945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:00.480687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.484105 (+ 3418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.484109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:00.484213 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:00.484237 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 03:17:00.486225 (+ 1988us) client_negotiation.cc:240] Negotiation successful
0929 03:17:00.486303 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6615,"thread_start_us":107,"threads_started":1}
I20260929 03:17:00.488250 25735 leader_election.cc:302] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 105e06477df0449395f88e408dee85e7, 5ffbcb61799d4a51adea1f502f994416; no voters:
I20260929 03:17:00.488495 25999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:00.468658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:47427 (local address 127.13.238.130:35463)
0929 03:17:00.468788 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:00.468793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:00.473738 (+ 4945us) server_negotiation.cc:407] Connection header received
0929 03:17:00.476544 (+ 2806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:00.476548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:00.476592 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:00.476640 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:00.477172 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.477975 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.486939 (+ 8964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.487443 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.487525 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:00.487566 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:00.488338 (+ 772us) server_negotiation.cc:299] Negotiation successful
0929 03:17:00.488408 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:17:00.488516 25649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417c9f14cc2944a5a58127fb94a6e373" candidate_uuid: "5ffbcb61799d4a51adea1f502f994416" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f31e16136494e59936a5cccfb4813d8" is_pre_election: true
I20260929 03:17:00.488663 26006 raft_consensus.cc:2781] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:17:00.488744 26006 raft_consensus.cc:482] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:17:00.488782 26006 raft_consensus.cc:3037] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:00.488819 25649 raft_consensus.cc:2454] T 417c9f14cc2944a5a58127fb94a6e373 P 7f31e16136494e59936a5cccfb4813d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ffbcb61799d4a51adea1f502f994416 in term 0.
I20260929 03:17:00.490119 26006 raft_consensus.cc:504] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105e06477df0449395f88e408dee85e7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 44413 } } peers { permanent_uuid: "7f31e16136494e59936a5cccfb4813d8" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 45617 } } peers { permanent_uuid: "5ffbcb61799d4a51adea1f502f994416" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35463 } }
I20260929 03:17:00.490969 25925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417c9f14cc2944a5a58127fb94a6e373" candidate_uuid: "5ffbcb61799d4a51adea1f502f994416" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "105e06477df0449395f88e408dee85e7"
I20260929 03:17:00.491087 25925 raft_consensus.cc:3037] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:00.492381 25925 raft_consensus.cc:2454] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ffbcb61799d4a51adea1f502f994416 in term 1.
I20260929 03:17:00.492784 25735 leader_election.cc:302] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 105e06477df0449395f88e408dee85e7, 5ffbcb61799d4a51adea1f502f994416; no voters:
I20260929 03:17:00.493517 25649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417c9f14cc2944a5a58127fb94a6e373" candidate_uuid: "5ffbcb61799d4a51adea1f502f994416" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f31e16136494e59936a5cccfb4813d8"
I20260929 03:17:00.493645 25649 raft_consensus.cc:3037] T 417c9f14cc2944a5a58127fb94a6e373 P 7f31e16136494e59936a5cccfb4813d8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:00.495090 25649 raft_consensus.cc:2454] T 417c9f14cc2944a5a58127fb94a6e373 P 7f31e16136494e59936a5cccfb4813d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ffbcb61799d4a51adea1f502f994416 in term 1.
I20260929 03:17:00.495499 26006 leader_election.cc:288] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416 [CANDIDATE]: Term 1 election: Requested vote from peers 105e06477df0449395f88e408dee85e7 (127.13.238.131:44413), 7f31e16136494e59936a5cccfb4813d8 (127.13.238.129:45617)
I20260929 03:17:00.495683 26006 raft_consensus.cc:2781] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:00.495821 26006 raft_consensus.cc:686] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416 [term 1 LEADER]: Becoming Leader. State: Replica: 5ffbcb61799d4a51adea1f502f994416, State: Running, Role: LEADER
I20260929 03:17:00.496230 26006 consensus_queue.cc:237] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105e06477df0449395f88e408dee85e7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 44413 } } peers { permanent_uuid: "7f31e16136494e59936a5cccfb4813d8" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 45617 } } peers { permanent_uuid: "5ffbcb61799d4a51adea1f502f994416" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35463 } }
I20260929 03:17:00.501144 25498 catalog_manager.cc:5495] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ffbcb61799d4a51adea1f502f994416 (127.13.238.130). New cstate: current_term: 1 leader_uuid: "5ffbcb61799d4a51adea1f502f994416" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105e06477df0449395f88e408dee85e7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 44413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f31e16136494e59936a5cccfb4813d8" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 45617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ffbcb61799d4a51adea1f502f994416" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35463 } health_report { overall_health: HEALTHY } } }
I20260929 03:17:00.503371 26012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:00.468334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:35463 (local address 127.13.238.131:47427)
0929 03:17:00.473661 (+ 5327us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:00.473675 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:00.475681 (+ 2006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:00.476893 (+ 1212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:00.476896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:00.476917 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:00.477045 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:00.477053 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.478160 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.478163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:00.479406 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:00.479417 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.487667 (+ 8250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.487670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:00.487771 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:00.487792 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 03:17:00.503048 (+ 15256us) client_negotiation.cc:240] Negotiation successful
0929 03:17:00.503174 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5259,"thread_start_us":89,"threads_started":1}
I20260929 03:17:00.506982 25984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:00.467276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:45617 (local address 127.13.238.131:42123)
0929 03:17:00.467541 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:00.467557 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:00.467653 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:00.481943 (+ 14290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:00.481947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:00.481969 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:00.482074 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:00.482081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.503679 (+ 21598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.503683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:00.504880 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:00.504895 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.506554 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.506558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:00.506690 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:00.506720 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 03:17:00.506776 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 03:17:00.506857 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188}
I20260929 03:17:00.507783 26011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:00.467842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:42123 (local address 127.13.238.129:45617)
0929 03:17:00.479731 (+ 11889us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:00.479740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:00.479755 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:17:00.479800 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:00.479803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:00.479871 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:00.482177 (+ 2306us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:00.482331 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.483084 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.505211 (+ 22127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.506417 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.507482 (+ 1065us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:00.507539 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:00.507576 (+ 37us) server_negotiation.cc:299] Negotiation successful
0929 03:17:00.507647 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11802,"thread_start_us":104,"threads_started":1}
I20260929 03:17:00.508253 25649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417c9f14cc2944a5a58127fb94a6e373" candidate_uuid: "105e06477df0449395f88e408dee85e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f31e16136494e59936a5cccfb4813d8" is_pre_election: true
I20260929 03:17:00.508463 25649 raft_consensus.cc:2379] T 417c9f14cc2944a5a58127fb94a6e373 P 7f31e16136494e59936a5cccfb4813d8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 105e06477df0449395f88e408dee85e7 in current term 1: Already voted for candidate 5ffbcb61799d4a51adea1f502f994416 in this term.
I20260929 03:17:00.508674 25793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417c9f14cc2944a5a58127fb94a6e373" candidate_uuid: "105e06477df0449395f88e408dee85e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ffbcb61799d4a51adea1f502f994416" is_pre_election: true
I20260929 03:17:00.509349 25869 leader_election.cc:302] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 105e06477df0449395f88e408dee85e7; no voters: 5ffbcb61799d4a51adea1f502f994416, 7f31e16136494e59936a5cccfb4813d8
I20260929 03:17:00.509918 26008 raft_consensus.cc:2726] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260929 03:17:00.527887 25983 tablet.cc:2369] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:17:00.536376 25713 tablet.cc:2369] T 417c9f14cc2944a5a58127fb94a6e373 P 7f31e16136494e59936a5cccfb4813d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:17:00.556983 25999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:00.551556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33486 (local address 127.13.238.130:35463)
0929 03:17:00.551674 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:00.551678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:00.551692 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:17:00.551876 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:00.551901 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:00.551974 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:00.552075 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:17:00.552442 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.553212 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.554702 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:00.555054 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:00.555161 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:00.555599 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:17:00.556225 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:17:00.556338 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:00.556769 (+ 431us) server_negotiation.cc:299] Negotiation successful
0929 03:17:00.556826 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 03:17:00 dist-test-slave-nbjg krb5kdc[25022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651819, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for HTTP/127.13.238.148@KRBTEST.COM
I20260929 03:17:00.564704 25649 raft_consensus.cc:1261] T 417c9f14cc2944a5a58127fb94a6e373 P 7f31e16136494e59936a5cccfb4813d8 [term 1 FOLLOWER]: Refusing update from remote peer 5ffbcb61799d4a51adea1f502f994416: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:00.565558 26013 consensus_queue.cc:1035] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f31e16136494e59936a5cccfb4813d8" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:00.578254 25925 raft_consensus.cc:1261] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7 [term 1 FOLLOWER]: Refusing update from remote peer 5ffbcb61799d4a51adea1f502f994416: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:00.578969 26013 consensus_queue.cc:1035] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416 [LEADER]: Connected to new peer: Peer: permanent_uuid: "105e06477df0449395f88e408dee85e7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 44413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:00.588673 26017 mvcc.cc:205] Tried to move back new op lower bound from 7334509857023799296 to 7334509856756453376. Current Snapshot: MvccSnapshot[applied={T|T < 7334509857023799296}]
I20260929 03:17:00.589377 26018 mvcc.cc:205] Tried to move back new op lower bound from 7334509857023799296 to 7334509856756453376. Current Snapshot: MvccSnapshot[applied={T|T < 7334509857023799296}]
I20260929 03:17:00.602782 25498 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:17:00.602989 25498 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:17:00.605999 25498 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7903e7509366430e9082f9118a8bf932: Sending DeleteTablet for 3 replicas of tablet 417c9f14cc2944a5a58127fb94a6e373
I20260929 03:17:00.607241 25644 tablet_service.cc:1514] Processing DeleteTablet for tablet 417c9f14cc2944a5a58127fb94a6e373 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:00 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:52736
I20260929 03:17:00.607820 25898 tablet_service.cc:1514] Processing DeleteTablet for tablet 417c9f14cc2944a5a58127fb94a6e373 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:00 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:50468
I20260929 03:17:00.608630 25770 tablet_service.cc:1514] Processing DeleteTablet for tablet 417c9f14cc2944a5a58127fb94a6e373 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:00 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:33480
I20260929 03:17:00.610095 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 25579
I20260929 03:17:00.626466 26026 tablet_replica.cc:330] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7: stopping tablet replica
I20260929 03:17:00.626775 26027 tablet_replica.cc:330] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416: stopping tablet replica
I20260929 03:17:00.627223 26027 raft_consensus.cc:2229] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:17:00.627604 26026 raft_consensus.cc:2229] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:17:00.628012 26026 raft_consensus.cc:2258] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:00.628413 26027 raft_consensus.cc:2258] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:00.631950 26027 ts_tablet_manager.cc:1905] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:00.635973 26026 ts_tablet_manager.cc:1905] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:00.638921 26026 ts_tablet_manager.cc:1918] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:17:00.643302 26027 ts_tablet_manager.cc:1918] T 417c9f14cc2944a5a58127fb94a6e373 P 5ffbcb61799d4a51adea1f502f994416: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260929 03:17:00.645073 25483 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.238.129:45617 (error 108)
I20260929 03:17:00.646760 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 25718
I20260929 03:17:00.648569 26026 log.cc:1196] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/417c9f14cc2944a5a58127fb94a6e373
I20260929 03:17:00.648998 26026 ts_tablet_manager.cc:1939] T 417c9f14cc2944a5a58127fb94a6e373 P 105e06477df0449395f88e408dee85e7: Deleting consensus metadata
I20260929 03:17:00.650736 25482 catalog_manager.cc:4841] TS 105e06477df0449395f88e408dee85e7 (127.13.238.131:44413): tablet 417c9f14cc2944a5a58127fb94a6e373 (table test-table [id=8d9c8898c6b74cd88fa8038e51ae97ae]) successfully deleted
I20260929 03:17:00.655254 25998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:00.654862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:45617 (local address 127.0.0.1:52738)
0929 03:17:00.655042 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:00.655143 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.129:45617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 03:17:00.655542 25483 catalog_manager.cc:4584] TS 7f31e16136494e59936a5cccfb4813d8 (127.13.238.129:45617): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 417c9f14cc2944a5a58127fb94a6e373: Network error: Client connection negotiation failed: client connection to 127.13.238.129:45617: connect: Connection refused (error 111)
I20260929 03:17:00.657141 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 25853
I20260929 03:17:00.663149 25998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:00.662740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:35463 (local address 127.0.0.1:33500)
0929 03:17:00.662978 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:00.663044 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.130:35463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260929 03:17:00.664552 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 25466
I20260929 03:17:00.673331 14266 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 03:17:01.008646 14266 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 03:17:01.008785 14266 mini_ranger.cc:66] Stopping Ranger...
I20260929 03:17:01.343389 14266 mini_ranger.cc:68] Stopped Ranger
2026-09-29 03:17:01.343 UTC [25052] LOG: received smart shutdown request
2026-09-29 03:17:01.347 UTC [25052] LOG: background worker "logical replication launcher" (PID 25061) exited with exit code 1
2026-09-29 03:17:01.347 UTC [25056] LOG: shutting down
2026-09-29 03:17:01.507 UTC [25052] LOG: database system is shut down
2026-09-29T03:17:01Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47484 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:17:01 dist-test-slave-nbjg krb5kdc[26039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:17:01 dist-test-slave-nbjg krb5kdc[26039](info): set up 2 sockets
Sep 29 03:17:01 dist-test-slave-nbjg krb5kdc[26039](info): commencing operation
krb5kdc: starting...
W20260929 03:17:02.562227 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:17:02 dist-test-slave-nbjg krb5kdc[26039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:17:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:17:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.13.238.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.13.238.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.13.238.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/rangeradmin_127.13.238.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.13.238.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/rangeradmin_127.13.238.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.13.238.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.13.238.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.13.238.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/rangerlookup_127.13.238.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.13.238.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/rangerlookup_127.13.238.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.238.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.238.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.238.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/HTTP_127.13.238.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.238.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/HTTP_127.13.238.148@KRBTEST.COM.keytab.
I20260929 03:17:02.728443 14266 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskdvlSPq/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/postgres -l logfile start
2026-09-29 03:17:05.148 UTC [26069] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 03:17:05.148 UTC [26069] LOG: listening on IPv4 address "127.13.238.148", port 48813
2026-09-29 03:17:05.151 UTC [26069] LOG: listening on Unix socket "/tmp/.s.PGSQL.48813"
2026-09-29 03:17:05.171 UTC [26072] LOG: database system was shut down at 2026-09-29 03:17:03 UTC
2026-09-29 03:17:05.175 UTC [26069] LOG: database system is ready to accept connections
I20260929 03:17:06.038080 14266 mini_postgres.cc:96] Postgres bound to 48813
2026-09-29 03:17:06.044 UTC [26080] FATAL: database "slave" does not exist
127.13.238.148:48813 - accepting connections
I20260929 03:17:06.044592 14266 mini_ranger.cc:161] Starting Ranger...
I20260929 03:17:06.059017 14266 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 03:17:06.516538 14266 mini_ranger.cc:87] Created ranger Postgres database
I20260929 03:17:06.516615 14266 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ranger-admin
2026-09-29 03:17:06,776 [I] DB FLAVOR :POSTGRES
2026-09-29 03:17:06,776 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 03:17:06,777 [I] Checking connection..
2026-09-29 03:17:06,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:48813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 03:17:07,022 [I] Checking connection passed.
2026-09-29 03:17:07,022 [I] --------- Verifying version history table ---------
2026-09-29 03:17:07,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:48813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:17:07,265 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 03:17:07,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:48813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:17:07,509 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 03:17:07,509 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 03:17:07,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:48813/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 03:17:07.765 UTC [26144] WARNING: there is no transaction in progress
2026-09-29 03:17:07,771 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 03:17:07,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:48813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:17:08,025 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 03:17:08,025 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 03:17:08,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:48813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 03:17:08,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:48813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 03:17:08,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:48813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-nbjg.c.gcp-upstream.internal','N') ;"
2026-09-29 03:17:08,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:48813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 03:17:09,009 [I] Table x_portal_user does not exist in database ranger
2026-09-29 03:17:09,009 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:48813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 03:17:09,261 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 03:17:09,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:48813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 03:17:09,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:48813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 03:17:09,752 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 03:17:09,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:48813/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 03:17:10.692 UTC [26252] WARNING: there is no transaction in progress
2026-09-29 03:17:10.709 UTC [26252] WARNING: there is no transaction in progress
2026-09-29 03:17:10.725 UTC [26252] WARNING: there is no transaction in progress
2026-09-29 03:17:10.740 UTC [26252] WARNING: there is no transaction in progress
2026-09-29 03:17:10.755 UTC [26252] WARNING: there is no transaction in progress
2026-09-29 03:17:10.776 UTC [26252] WARNING: there is no transaction in progress
2026-09-29 03:17:10.834 UTC [26252] WARNING: there is no transaction in progress
2026-09-29 03:17:10.841 UTC [26252] WARNING: there is no transaction in progress
2026-09-29 03:17:10.856 UTC [26252] WARNING: there is no transaction in progress
2026-09-29 03:17:10.868 UTC [26252] WARNING: there is no transaction in progress
2026-09-29 03:17:10.879 UTC [26252] WARNING: there is no transaction in progress
2026-09-29 03:17:10.893 UTC [26252] WARNING: there is no transaction in progress
2026-09-29 03:17:10.903 UTC [26252] WARNING: there is no transaction in progress
2026-09-29 03:17:10.909 UTC [26252] WARNING: there is no transaction in progress
2026-09-29 03:17:10.916 UTC [26252] WARNING: there is no transaction in progress
2026-09-29 03:17:10.924 UTC [26252] WARNING: there is no transaction in progress
2026-09-29 03:17:11,572 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 03:17:11,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:48813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 03:17:11,802 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 03:17:11,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:48813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 03:17:12,060 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 03:17:12,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:48813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 03:17:12,288 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 03:17:12,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:48813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 03:17:12,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:48813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 03:17:12,752 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:48813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-nbjg.c.gcp-upstream.internal';"
2026-09-29 03:17:12,998 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 03:17:12,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:48813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 03:17:13,255 [I] DB_PATCHES have already been applied
I20260929 03:17:13.260840 14266 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ranger-admin:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskdvlSPq/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 03:17:13.260912 14266 mini_ranger.cc:193] Using host: 127.13.238.148
I20260929 03:17:13.264163 14266 mini_ranger.cc:227] Ranger admin URL: http://127.13.238.148:40571
Sep 29, 2026 3:17:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:17:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:17:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 3:17:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 3:17:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 3:17:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 3:17:14 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 03:17:14.251588 14266 mini_ranger.cc:160] Time spent starting Ranger: real 8.207s user 0.002s sys 0.006s
Sep 29, 2026 3:17:34 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 3:17:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 3:17:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 3:17:34 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 3:17:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 3:17:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 3:17:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 3:17:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 3:17:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 3:17:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 3:17:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 3:17:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 3:17:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 3:17:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 3:17:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 3:17:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 3:17:34 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 3:17:35 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 03:17:35.874818 14266 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.13.238.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.13.238.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.13.238.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/rangerkms_127.13.238.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.13.238.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/rangerkms_127.13.238.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.238.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.13.238.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.238.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/HTTP_127.13.238.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.238.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/HTTP_127.13.238.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 03:17:35 dist-test-slave-nbjg krb5kdc[26039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651855, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 03:17:35.955030 14266 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 03:17:35.964557 14266 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 03:17:36.812410 14266 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 03:17:36.812489 14266 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ranger-kms
I20260929 03:17:36.812505 14266 mini_ranger_kms.cc:227] Using postgres at 127.13.238.148:48813
2026-09-29 03:17:36,966 [I] DB FLAVOR :POSTGRES
2026-09-29 03:17:36,966 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 03:17:36,966 [I] Checking connection
2026-09-29 03:17:36,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:48813/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 03:17:37,192 [I] connection success
2026-09-29 03:17:37,192 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 03:17:37,192 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 03:17:37,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:48813/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 03:17:37,422 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 03:17:37,422 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 03:17:37,422 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 03:17:37,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.238.148:48813/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 03:17:37,673 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 03:17:38.111083 14266 mini_ranger_kms.cc:310] Created kms service
I20260929 03:17:38.298664 14266 mini_ranger_kms.cc:326] Created kudu user
I20260929 03:17:38.343042 14266 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 03:17:38 dist-test-slave-nbjg krb5kdc[26039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651855, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.13.238.148@KRBTEST.COM
I20260929 03:17:38.736857 14266 mini_ranger_kms.cc:384] Added ranger policy
I20260929 03:17:38.737010 14266 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ranger-kms:/tmp/dist-test-taskdvlSPq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskdvlSPq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 03:17:38.737051 14266 mini_ranger_kms.cc:240] Using host: 127.13.238.148
I20260929 03:17:38.740904 14266 mini_ranger_kms.cc:276] Ranger KMS PID: 26432
I20260929 03:17:38.740998 14266 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.13.238.148:52131
Sep 29, 2026 3:17:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:17:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:17:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 3:17:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 3:17:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 3:17:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 03:17:40.772578 14266 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.818s user 0.004s sys 0.135s
I20260929 03:17:40.772686 14266 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 03:17:40.772768 14266 mini_ranger_kms.cc:401] 127.13.238.148:52131/kms/v1/keys
Sep 29 03:17:40 dist-test-slave-nbjg krb5kdc[26039](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790651860, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.238.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 03:17:41 dist-test-slave-nbjg krb5kdc[26039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651860, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.238.148@KRBTEST.COM for HTTP/127.13.238.148@KRBTEST.COM
Sep 29 03:17:41 dist-test-slave-nbjg krb5kdc[26039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651860, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.238.148@KRBTEST.COM for HTTP/127.13.238.148@KRBTEST.COM
Sep 29 03:17:43 dist-test-slave-nbjg krb5kdc[26039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:17:43.737224 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:46833
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:35591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:46833
--ranger_config_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.238.148:52131/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:43.846226 26483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:43.846520 26483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:43.846563 26483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:43.850553 26483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:43.850611 26483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:43.850629 26483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:43.850644 26483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:43.850660 26483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:17:43.856140 26483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:35591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/wal
--ranger_kms_url=127.13.238.148:52131/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:46833
--ranger_config_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:46833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:17:43.857254 26483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:43.858320 26483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:43.866492 26488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:43.866564 26491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:43.866726 26483 server_base.cc:1034] running on GCE node
W20260929 03:17:43.866734 26489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:43.867334 26483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:43.868391 26483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:43.869549 26483 hybrid_clock.cc:648] HybridClock initialized: now 1790651863869519 us; error 46 us; skew 500 ppm
Sep 29 03:17:43 dist-test-slave-nbjg krb5kdc[26039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651863, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:43.875419 26483 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:17:43.876363 26483 webserver.cc:460] Webserver started at http://127.13.238.190:41849/ using document root <none> and password file <none>
I20260929 03:17:43.876886 26483 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:43.876937 26483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:43.877146 26483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:17:43 dist-test-slave-nbjg krb5kdc[26039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651863, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for HTTP/127.13.238.148@KRBTEST.COM
I20260929 03:17:43.935428 26483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "2c4242b6615d44b7bd49150ab213ac10"
format_stamp: "Formatted at 2026-09-29 03:17:43 on dist-test-slave-nbjg"
server_key: "4d5cc141aa78ebf2e823de222aad473f"
server_key_iv: "9dae4aa08e8fcfb64325e8c4cfdcb4da"
server_key_version: "kuduclusterkey@0"
I20260929 03:17:43.936017 26483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "2c4242b6615d44b7bd49150ab213ac10"
format_stamp: "Formatted at 2026-09-29 03:17:43 on dist-test-slave-nbjg"
server_key: "4d5cc141aa78ebf2e823de222aad473f"
server_key_iv: "9dae4aa08e8fcfb64325e8c4cfdcb4da"
server_key_version: "kuduclusterkey@0"
I20260929 03:17:43.939945 26483 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:17:43.954061 26498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:43.955446 26483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:17:43.955571 26483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "2c4242b6615d44b7bd49150ab213ac10"
format_stamp: "Formatted at 2026-09-29 03:17:43 on dist-test-slave-nbjg"
server_key: "4d5cc141aa78ebf2e823de222aad473f"
server_key_iv: "9dae4aa08e8fcfb64325e8c4cfdcb4da"
server_key_version: "kuduclusterkey@0"
I20260929 03:17:43.955698 26483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:43.969307 26483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:43.972082 26483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:43.972261 26483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:43.979681 26483 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:46833
I20260929 03:17:43.979733 26550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:46833 every 8 connection(s)
I20260929 03:17:43.980643 26483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:17:43.984200 26551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:43.984859 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 26483
I20260929 03:17:43.984968 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:17:43.985270 14266 external_mini_cluster.cc:1428] Setting key 6776eb6b8052c1d8c209f40800876d15
I20260929 03:17:43.991927 26551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10: Bootstrap starting.
Sep 29 03:17:43 dist-test-slave-nbjg krb5kdc[26039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:17:43.994678 26551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:43.995929 26551 log.cc:824] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:43.998257 26551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10: No bootstrap required, opened a new log
I20260929 03:17:43.998376 26554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:43.986697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36264 (local address 127.13.238.190:46833)
0929 03:17:43.987091 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:43.987110 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:43.987146 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:17:43.987811 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:43.987827 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:43.988156 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:43.988462 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:43.989222 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:43.990232 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:43.991033 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:43.991498 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:43.991638 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:43.993933 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:43.993938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:43.993951 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:43.993977 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:43.995502 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:43.995915 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:43.995920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:43.995925 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:43.996054 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:43.996328 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:43.996332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:43.996334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:43.996824 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:43.997018 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:43.997304 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 03:17:43.997588 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":109,"threads_started":1}
I20260929 03:17:44.002324 26551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4242b6615d44b7bd49150ab213ac10" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 46833 } }
I20260929 03:17:44.002614 26551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:44.002691 26551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4242b6615d44b7bd49150ab213ac10, State: Initialized, Role: FOLLOWER
I20260929 03:17:44.003158 26551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4242b6615d44b7bd49150ab213ac10" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 46833 } }
I20260929 03:17:44.003316 26551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:17:44.003446 26551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:17:44.003556 26551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:44.004664 26551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4242b6615d44b7bd49150ab213ac10" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 46833 } }
I20260929 03:17:44.004963 26551 leader_election.cc:302] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c4242b6615d44b7bd49150ab213ac10; no voters:
I20260929 03:17:44.005214 26551 leader_election.cc:288] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:17:44.005296 26556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:44.005538 26556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [term 1 LEADER]: Becoming Leader. State: Replica: 2c4242b6615d44b7bd49150ab213ac10, State: Running, Role: LEADER
I20260929 03:17:44.005864 26556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4242b6615d44b7bd49150ab213ac10" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 46833 } }
I20260929 03:17:44.006387 26551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:44.007480 26558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c4242b6615d44b7bd49150ab213ac10. Latest consensus state: current_term: 1 leader_uuid: "2c4242b6615d44b7bd49150ab213ac10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4242b6615d44b7bd49150ab213ac10" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 46833 } } }
I20260929 03:17:44.007599 26558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:44.007905 26557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c4242b6615d44b7bd49150ab213ac10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4242b6615d44b7bd49150ab213ac10" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 46833 } } }
I20260929 03:17:44.007988 26557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:44.008311 26565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:17:44.008421 26551 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 03:17:44.013551 26565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:17:44.020193 26565 catalog_manager.cc:1347] Generated new cluster ID: 02a06e56acd74490ab78f1f313f8ae31
I20260929 03:17:44.020255 26565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:17:44.030048 26565 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:17:44.031452 26565 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:17:44.047004 26565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10: Generated new TSK 0
I20260929 03:17:44.047845 26565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
I20260929 03:17:44.705080 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.238.148:52131/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:0
--local_ip_for_outbound_sockets=127.13.238.129
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:46833
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:35591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:44.827950 26596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:44.828179 26596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:44.828219 26596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:44.832229 26596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:44.832285 26596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:44.832361 26596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:17:44.837718 26596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:35591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.13.238.148:52131/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:46833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:17:44.838759 26596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:44.839780 26596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:44.851773 26602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:44.851781 26603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:44.852315 26605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:44.854087 26596 server_base.cc:1034] running on GCE node
I20260929 03:17:44.854562 26596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:44.855269 26596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:44.856431 26596 hybrid_clock.cc:648] HybridClock initialized: now 1790651864856393 us; error 57 us; skew 500 ppm
Sep 29 03:17:44 dist-test-slave-nbjg krb5kdc[26039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651864, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:44.859567 26596 init.cc:377] Logged in from keytab as kudu/127.13.238.129@KRBTEST.COM (short username kudu)
I20260929 03:17:44.860567 26596 webserver.cc:460] Webserver started at http://127.13.238.129:40917/ using document root <none> and password file <none>
I20260929 03:17:44.861153 26596 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:44.861231 26596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:44.861457 26596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:17:44 dist-test-slave-nbjg krb5kdc[26039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651864, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for HTTP/127.13.238.148@KRBTEST.COM
I20260929 03:17:44.878476 26596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/data/instance:
uuid: "5ec3f493ee5345b9aac606ba82973364"
format_stamp: "Formatted at 2026-09-29 03:17:44 on dist-test-slave-nbjg"
server_key: "0727c29aa3c19088d461be2d57b5fb93"
server_key_iv: "d56e4a3db74af21ee4b8dbacb7aa9bc3"
server_key_version: "kuduclusterkey@0"
I20260929 03:17:44.879233 26596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance:
uuid: "5ec3f493ee5345b9aac606ba82973364"
format_stamp: "Formatted at 2026-09-29 03:17:44 on dist-test-slave-nbjg"
server_key: "0727c29aa3c19088d461be2d57b5fb93"
server_key_iv: "d56e4a3db74af21ee4b8dbacb7aa9bc3"
server_key_version: "kuduclusterkey@0"
I20260929 03:17:44.883229 26596 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 03:17:44.901526 26613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:44.903446 26596 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260929 03:17:44.903621 26596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "5ec3f493ee5345b9aac606ba82973364"
format_stamp: "Formatted at 2026-09-29 03:17:44 on dist-test-slave-nbjg"
server_key: "0727c29aa3c19088d461be2d57b5fb93"
server_key_iv: "d56e4a3db74af21ee4b8dbacb7aa9bc3"
server_key_version: "kuduclusterkey@0"
I20260929 03:17:44.903781 26596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:44.919826 26596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:44.936823 26596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:44.937130 26596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:44.937983 26596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:44.939179 26596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:44.947635 26596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.000s sys 0.000s
I20260929 03:17:44.947803 26596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:44.947870 26596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:44.957623 26596 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:40163
I20260929 03:17:44.957683 26728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:40163 every 8 connection(s)
I20260929 03:17:44.958643 26596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:17:44.963743 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 26596
I20260929 03:17:44.963872 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance
I20260929 03:17:44.964154 14266 external_mini_cluster.cc:1428] Setting key 2d0de8b089ebbaa2fe4b94077d9fd1b9
Sep 29 03:17:44 dist-test-slave-nbjg krb5kdc[26039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651864, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:17:44.975217 26731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:44.960880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:42847 (local address 127.13.238.190:46833)
0929 03:17:44.961203 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:44.961215 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:44.961986 (+ 771us) server_negotiation.cc:407] Connection header received
0929 03:17:44.963192 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:44.963196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:44.963255 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:44.963377 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:44.964755 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:44.965631 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:44.967390 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:44.967760 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:44.967832 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:44.970830 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:44.970835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:44.970837 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:44.970862 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:44.972665 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:44.973374 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:44.973378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:44.973380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:44.973468 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:44.973858 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:44.973862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:44.973864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:44.974102 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:44.974183 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:44.974880 (+ 697us) server_negotiation.cc:299] Negotiation successful
0929 03:17:44.975010 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for kudu/127.13.238.130@KRBTEST.COM; defaulting to no policy
I20260929 03:17:44.977207 26732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:44.961309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:46833 (local address 127.13.238.129:42847)
0929 03:17:44.961803 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:44.961862 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:44.962939 (+ 1077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:44.963557 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:44.963567 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:44.963909 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:44.964586 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:44.964607 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:44.965794 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:44.965797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:44.967253 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:44.967264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:44.968034 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:44.968037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:44.968243 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:44.968905 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:44.968926 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:44.970630 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:44.972807 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:44.972813 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:44.972833 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:44.973257 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:44.973570 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:44.973573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:44.973575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:44.973753 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:44.974268 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:44.974274 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:44.974599 (+ 325us) client_negotiation.cc:769] Sending connection context
0929 03:17:44.975683 (+ 1084us) client_negotiation.cc:240] Negotiation successful
0929 03:17:44.976048 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":102,"threads_started":1}
I20260929 03:17:44.978711 26729 heartbeater.cc:347] Connected to a master server at 127.13.238.190:46833
I20260929 03:17:44.978976 26729 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.130@KRBTEST.COM" created.
I20260929 03:17:44.979601 26729 heartbeater.cc:511] Master 127.13.238.190:46833 requested a full tablet report, sending...
I20260929 03:17:44.982594 26512 ts_manager.cc:194] Registered new tserver with Master: 5ec3f493ee5345b9aac606ba82973364 (127.13.238.129:40163)
I20260929 03:17:44.984050 26512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.129@KRBTEST.COM'} at 127.13.238.129:42847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
I20260929 03:17:45.005000 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.238.148:52131/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:0
--local_ip_for_outbound_sockets=127.13.238.130
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:46833
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:35591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:17:45 dist-test-slave-nbjg krb5kdc[26039](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790651865, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 03:17:45.139024 26735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:45.139398 26735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:45.139484 26735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:45.144484 26735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:45.144550 26735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:45.144629 26735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:17:45.150270 26735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:35591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.13.238.148:52131/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:46833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:17:45.151320 26735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:45.152374 26735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:45.161840 26741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:45.161834 26744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:45.161836 26742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:45.162326 26735 server_base.cc:1034] running on GCE node
I20260929 03:17:45.162694 26735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:45.163321 26735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:45.164611 26735 hybrid_clock.cc:648] HybridClock initialized: now 1790651865164570 us; error 58 us; skew 500 ppm
Sep 29 03:17:45 dist-test-slave-nbjg krb5kdc[26039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651865, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:45.167198 26735 init.cc:377] Logged in from keytab as kudu/127.13.238.130@KRBTEST.COM (short username kudu)
I20260929 03:17:45.168079 26735 webserver.cc:460] Webserver started at http://127.13.238.130:43339/ using document root <none> and password file <none>
I20260929 03:17:45.168609 26735 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:45.168684 26735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:45.168906 26735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:17:45 dist-test-slave-nbjg krb5kdc[26039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651865, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for HTTP/127.13.238.148@KRBTEST.COM
I20260929 03:17:45.192094 26735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/data/instance:
uuid: "5a65c74e75cf4b43af0ccde9d37b0fae"
format_stamp: "Formatted at 2026-09-29 03:17:45 on dist-test-slave-nbjg"
server_key: "768de0f69854101e0207959a5005a537"
server_key_iv: "576719b04b09dda698e12864aa1d2940"
server_key_version: "kuduclusterkey@0"
I20260929 03:17:45.193912 26735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance:
uuid: "5a65c74e75cf4b43af0ccde9d37b0fae"
format_stamp: "Formatted at 2026-09-29 03:17:45 on dist-test-slave-nbjg"
server_key: "768de0f69854101e0207959a5005a537"
server_key_iv: "576719b04b09dda698e12864aa1d2940"
server_key_version: "kuduclusterkey@0"
I20260929 03:17:45.209981 26735 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.003s sys 0.000s
I20260929 03:17:45.236498 26751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:45.238353 26735 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260929 03:17:45.238489 26735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "5a65c74e75cf4b43af0ccde9d37b0fae"
format_stamp: "Formatted at 2026-09-29 03:17:45 on dist-test-slave-nbjg"
server_key: "768de0f69854101e0207959a5005a537"
server_key_iv: "576719b04b09dda698e12864aa1d2940"
server_key_version: "kuduclusterkey@0"
I20260929 03:17:45.238622 26735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:45.264003 26735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:45.269745 26735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:45.270181 26735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:45.271552 26735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:45.273414 26735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:45.273559 26735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:45.273658 26735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:45.273757 26735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:45.285177 26735 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:40965
I20260929 03:17:45.286420 26735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:17:45.287243 26864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:40965 every 8 connection(s)
I20260929 03:17:45.290526 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 26735
I20260929 03:17:45.290673 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance
I20260929 03:17:45.290987 14266 external_mini_cluster.cc:1428] Setting key 5ca7cadcb27e3a34282dbfb07a2f8f1d
WARNING: no policy specified for kudu/127.13.238.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.131@KRBTEST.COM" created.
Sep 29 03:17:45 dist-test-slave-nbjg krb5kdc[26039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651865, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:17:45.331151 26731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:45.290464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:34245 (local address 127.13.238.190:46833)
0929 03:17:45.290618 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:45.290622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:45.308360 (+ 17738us) server_negotiation.cc:407] Connection header received
0929 03:17:45.309710 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:45.309714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:45.309775 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:45.309872 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:45.319787 (+ 9915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:45.320652 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:45.323409 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:45.323772 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:45.323884 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:45.327393 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:45.327397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:45.327400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:45.327430 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:45.329097 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:45.329598 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:45.329600 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:45.329602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:45.329684 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:45.330017 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:45.330020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:45.330022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:45.330230 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:45.330322 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:45.330832 (+ 510us) server_negotiation.cc:299] Negotiation successful
0929 03:17:45.330960 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:17:45.332017 26867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:45.289748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:46833 (local address 127.13.238.130:34245)
0929 03:17:45.308134 (+ 18386us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:45.308222 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:45.309429 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:45.318385 (+ 8956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:45.318399 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:45.318809 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:45.319543 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:45.319564 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:45.322285 (+ 2721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:45.322288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:45.323271 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:45.323282 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:45.324095 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:45.324100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:45.324342 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:45.325159 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:45.325197 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:45.327162 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:45.329221 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:45.329225 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:45.329240 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:45.329484 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:45.329766 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:45.329770 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:45.329771 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:45.329917 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:45.330323 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:45.330331 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:45.330589 (+ 258us) client_negotiation.cc:769] Sending connection context
0929 03:17:45.330757 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 03:17:45.331076 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18016,"thread_start_us":225,"threads_started":1}
I20260929 03:17:45.333206 26865 heartbeater.cc:347] Connected to a master server at 127.13.238.190:46833
I20260929 03:17:45.333451 26865 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:45.333941 26865 heartbeater.cc:511] Master 127.13.238.190:46833 requested a full tablet report, sending...
I20260929 03:17:45.335064 26512 ts_manager.cc:194] Registered new tserver with Master: 5a65c74e75cf4b43af0ccde9d37b0fae (127.13.238.130:40965)
I20260929 03:17:45.335759 26512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.130@KRBTEST.COM'} at 127.13.238.130:34245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
I20260929 03:17:45.342007 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.238.148:52131/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:0
--local_ip_for_outbound_sockets=127.13.238.131
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:46833
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:35591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:17:45 dist-test-slave-nbjg krb5kdc[26039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651865, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for HTTP/127.13.238.148@KRBTEST.COM
W20260929 03:17:45.499573 26870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:45.499822 26870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:45.499881 26870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:45.505172 26870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:45.505236 26870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:45.505317 26870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:17:45.511158 26870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:35591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.13.238.148:52131/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:46833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:17:45.512243 26870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:45.513499 26870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:45.527482 26876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:45.527833 26875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:45.527882 26878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:45.528872 26870 server_base.cc:1034] running on GCE node
I20260929 03:17:45.529371 26870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:45.530011 26870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:45.531212 26870 hybrid_clock.cc:648] HybridClock initialized: now 1790651865531142 us; error 95 us; skew 500 ppm
Sep 29 03:17:45 dist-test-slave-nbjg krb5kdc[26039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651865, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:45.534870 26870 init.cc:377] Logged in from keytab as kudu/127.13.238.131@KRBTEST.COM (short username kudu)
I20260929 03:17:45.536010 26870 webserver.cc:460] Webserver started at http://127.13.238.131:45607/ using document root <none> and password file <none>
I20260929 03:17:45.536762 26870 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:45.536880 26870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:45.537145 26870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:17:45 dist-test-slave-nbjg krb5kdc[26039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651865, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for HTTP/127.13.238.148@KRBTEST.COM
I20260929 03:17:45.572430 26870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/data/instance:
uuid: "444f2fdba3644bbb8c971717fa919ba9"
format_stamp: "Formatted at 2026-09-29 03:17:45 on dist-test-slave-nbjg"
server_key: "fcd71039da1c1f8e77e60072ce6db300"
server_key_iv: "6fa0e921539a47d7baabbc002ebaeeb6"
server_key_version: "kuduclusterkey@0"
I20260929 03:17:45.574364 26870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance:
uuid: "444f2fdba3644bbb8c971717fa919ba9"
format_stamp: "Formatted at 2026-09-29 03:17:45 on dist-test-slave-nbjg"
server_key: "fcd71039da1c1f8e77e60072ce6db300"
server_key_iv: "6fa0e921539a47d7baabbc002ebaeeb6"
server_key_version: "kuduclusterkey@0"
I20260929 03:17:45.579882 26870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 03:17:45.610656 26885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:45.612515 26870 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.003s sys 0.000s
I20260929 03:17:45.612653 26870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "444f2fdba3644bbb8c971717fa919ba9"
format_stamp: "Formatted at 2026-09-29 03:17:45 on dist-test-slave-nbjg"
server_key: "fcd71039da1c1f8e77e60072ce6db300"
server_key_iv: "6fa0e921539a47d7baabbc002ebaeeb6"
server_key_version: "kuduclusterkey@0"
I20260929 03:17:45.612797 26870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:45.656410 26870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:45.659842 26870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:45.660063 26870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:45.660702 26870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:45.661687 26870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:45.661772 26870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:45.661847 26870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:45.661891 26870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:45.671271 26870 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:36927
I20260929 03:17:45.672503 26870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:17:45.673220 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 26870
I20260929 03:17:45.673342 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance
I20260929 03:17:45.673643 14266 external_mini_cluster.cc:1428] Setting key d6fd3a13f03635a45dcc2a58e447992a
I20260929 03:17:45.685201 26998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:36927 every 8 connection(s)
Sep 29 03:17:45 dist-test-slave-nbjg krb5kdc[26039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651865, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:17:45.701761 26731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:45.685953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:41137 (local address 127.13.238.190:46833)
0929 03:17:45.686097 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:45.686102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:45.687680 (+ 1578us) server_negotiation.cc:407] Connection header received
0929 03:17:45.688883 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:45.688886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:45.688939 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:45.688988 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:45.690686 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:45.691511 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:45.693209 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:45.693542 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:45.693622 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:45.696709 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:45.696713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:45.696715 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:45.696742 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:45.699466 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:45.700054 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:45.700058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:45.700060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:45.700141 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:45.700531 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:45.700535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:45.700537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:45.700742 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:45.700824 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:45.701485 (+ 661us) server_negotiation.cc:299] Negotiation successful
0929 03:17:45.701599 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:17:45.703593 27001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:45.686264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:46833 (local address 127.13.238.131:41137)
0929 03:17:45.687531 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:45.687589 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:45.688642 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:45.689414 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:45.689423 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:45.689791 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:45.690516 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:45.690539 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:45.691667 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:45.691671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:45.693063 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:45.693075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:45.693829 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:45.693833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:45.694030 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:45.694710 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:45.694732 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:45.696519 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:45.699588 (+ 3069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:45.699595 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:45.699611 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:45.699941 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:45.700257 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:45.700260 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:45.700262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:45.700431 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:45.700889 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:45.700896 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:45.701205 (+ 309us) client_negotiation.cc:769] Sending connection context
0929 03:17:45.702172 (+ 967us) client_negotiation.cc:240] Negotiation successful
0929 03:17:45.702524 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1035,"thread_start_us":167,"threads_started":1}
I20260929 03:17:45.705073 26999 heartbeater.cc:347] Connected to a master server at 127.13.238.190:46833
I20260929 03:17:45.705333 26999 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:45.705893 26999 heartbeater.cc:511] Master 127.13.238.190:46833 requested a full tablet report, sending...
I20260929 03:17:45.707095 26512 ts_manager.cc:194] Registered new tserver with Master: 444f2fdba3644bbb8c971717fa919ba9 (127.13.238.131:36927)
I20260929 03:17:45.707858 26512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.131@KRBTEST.COM'} at 127.13.238.131:41137
I20260929 03:17:45.711901 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:17:45.712615 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 26596
I20260929 03:17:45.722010 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 26735
I20260929 03:17:45.730043 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 26870
I20260929 03:17:45.736200 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 26483
I20260929 03:17:45.744450 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:46833
--webserver_interface=127.13.238.190
--webserver_port=41849
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:35591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:46833
--ranger_config_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.238.148:52131/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.238.148:52131/kms with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:45.853667 27002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:45.853919 27002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:45.853956 27002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:45.857934 27002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:45.857986 27002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:45.858006 27002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:45.858019 27002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:45.858033 27002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:17:45.863468 27002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:35591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.238.148:52131/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:46833
--ranger_config_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:46833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=41849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:17:45.864492 27002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:45.865484 27002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:45.873687 27010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:45.873697 27007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:45.873988 27002 server_base.cc:1034] running on GCE node
W20260929 03:17:45.874219 27008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:45.874766 27002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:45.875938 27002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:45.877136 27002 hybrid_clock.cc:648] HybridClock initialized: now 1790651865877114 us; error 30 us; skew 500 ppm
Sep 29 03:17:45 dist-test-slave-nbjg krb5kdc[26039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651865, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:45.879789 27002 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:17:45.880662 27002 webserver.cc:460] Webserver started at http://127.13.238.190:41849/ using document root <none> and password file <none>
I20260929 03:17:45.881174 27002 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:45.881235 27002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 03:17:45 dist-test-slave-nbjg krb5kdc[26039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651865, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for HTTP/127.13.238.148@KRBTEST.COM
I20260929 03:17:45.939899 27002 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 03:17:45.942591 27018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:45.943934 27002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:17:45.944046 27002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "2c4242b6615d44b7bd49150ab213ac10"
format_stamp: "Formatted at 2026-09-29 03:17:43 on dist-test-slave-nbjg"
server_key: "4d5cc141aa78ebf2e823de222aad473f"
server_key_iv: "9dae4aa08e8fcfb64325e8c4cfdcb4da"
server_key_version: "kuduclusterkey@0"
I20260929 03:17:45.944589 27002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:45.971369 27002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:45.973814 27002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:45.973977 27002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:45.980558 27002 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:46833
I20260929 03:17:45.980572 27070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:46833 every 8 connection(s)
I20260929 03:17:45.981555 27002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:17:45.985630 27071 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:17:45.987223 27071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10: Bootstrap starting.
I20260929 03:17:45.991498 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 27002
I20260929 03:17:45.991991 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.238.148:52131/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:40163
--local_ip_for_outbound_sockets=127.13.238.129
--tserver_master_addrs=127.13.238.190:46833
--webserver_port=40917
--webserver_interface=127.13.238.129
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:35591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.238.148:52131/kms with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:17:46.003899 27071 log.cc:824] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:46.011051 27071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:17:46.011421 27071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10: Bootstrap complete.
I20260929 03:17:46.015341 27071 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4242b6615d44b7bd49150ab213ac10" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 46833 } }
I20260929 03:17:46.015775 27071 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4242b6615d44b7bd49150ab213ac10, State: Initialized, Role: FOLLOWER
I20260929 03:17:46.016283 27071 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4242b6615d44b7bd49150ab213ac10" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 46833 } }
I20260929 03:17:46.016405 27071 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:17:46.016471 27071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:17:46.016562 27071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:17:46.018491 27071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4242b6615d44b7bd49150ab213ac10" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 46833 } }
I20260929 03:17:46.018805 27071 leader_election.cc:302] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c4242b6615d44b7bd49150ab213ac10; no voters:
I20260929 03:17:46.019080 27071 leader_election.cc:288] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 03:17:46.019232 27075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:17:46.019574 27075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [term 2 LEADER]: Becoming Leader. State: Replica: 2c4242b6615d44b7bd49150ab213ac10, State: Running, Role: LEADER
I20260929 03:17:46.019933 27075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4242b6615d44b7bd49150ab213ac10" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 46833 } }
I20260929 03:17:46.020259 27071 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:46.021308 27076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2c4242b6615d44b7bd49150ab213ac10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4242b6615d44b7bd49150ab213ac10" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 46833 } } }
I20260929 03:17:46.021427 27076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:46.022184 27077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c4242b6615d44b7bd49150ab213ac10. Latest consensus state: current_term: 2 leader_uuid: "2c4242b6615d44b7bd49150ab213ac10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4242b6615d44b7bd49150ab213ac10" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 46833 } } }
I20260929 03:17:46.022363 27077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:46.022493 27084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:17:46.022464 27071 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 03:17:46.028434 27084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:17:46.029323 27084 catalog_manager.cc:1259] Loaded cluster ID: 02a06e56acd74490ab78f1f313f8ae31
I20260929 03:17:46.029417 27084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:17:46.031950 27084 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:17:46.032706 27084 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10: Loaded TSK: 0
I20260929 03:17:46.034022 27084 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:17:46.139812 27073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:46.140189 27073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:46.140259 27073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:46.146320 27073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:46.146440 27073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:46.146554 27073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:17:46.154397 27073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:35591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.238.148:52131/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:40163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=40917
--tserver_master_addrs=127.13.238.190:46833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:17:46.155819 27073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:46.157425 27073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:46.169786 27108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:46.170214 27073 server_base.cc:1034] running on GCE node
W20260929 03:17:46.170320 27111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:46.170554 27109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:46.171039 27073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:46.171809 27073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:46.173017 27073 hybrid_clock.cc:648] HybridClock initialized: now 1790651866172990 us; error 37 us; skew 500 ppm
Sep 29 03:17:46 dist-test-slave-nbjg krb5kdc[26039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651866, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:46.176612 27073 init.cc:377] Logged in from keytab as kudu/127.13.238.129@KRBTEST.COM (short username kudu)
I20260929 03:17:46.177752 27073 webserver.cc:460] Webserver started at http://127.13.238.129:40917/ using document root <none> and password file <none>
I20260929 03:17:46.178594 27073 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:46.178771 27073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 03:17:46 dist-test-slave-nbjg krb5kdc[26039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651866, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for HTTP/127.13.238.148@KRBTEST.COM
I20260929 03:17:46.203672 27073 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.000s
I20260929 03:17:46.207787 27119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:46.211153 27073 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260929 03:17:46.211282 27073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "5ec3f493ee5345b9aac606ba82973364"
format_stamp: "Formatted at 2026-09-29 03:17:44 on dist-test-slave-nbjg"
server_key: "0727c29aa3c19088d461be2d57b5fb93"
server_key_iv: "d56e4a3db74af21ee4b8dbacb7aa9bc3"
server_key_version: "kuduclusterkey@0"
I20260929 03:17:46.211882 27073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:46.235812 27073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:46.239568 27073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:46.239825 27073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:46.240595 27073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:46.241708 27073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:46.241814 27073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:46.241947 27073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:46.242031 27073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:46.253660 27073 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:40163
I20260929 03:17:46.253705 27232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:40163 every 8 connection(s)
I20260929 03:17:46.254771 27073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:17:46.259517 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 27073
I20260929 03:17:46.259980 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.238.148:52131/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:40965
--local_ip_for_outbound_sockets=127.13.238.130
--tserver_master_addrs=127.13.238.190:46833
--webserver_port=43339
--webserver_interface=127.13.238.130
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:35591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.238.148:52131/kms with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:17:46 dist-test-slave-nbjg krb5kdc[26039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651866, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:17:46.274771 27235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:46.257246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:46833 (local address 127.13.238.129:45801)
0929 03:17:46.257746 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:46.257798 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:46.258795 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:46.260850 (+ 2055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:46.260859 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:46.261180 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:46.261813 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:46.261832 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:46.263462 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:46.263465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:46.264715 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:46.264727 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:46.265961 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:46.265965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:46.266156 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:46.266853 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:46.266874 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:46.268734 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:46.271237 (+ 2503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:46.271243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:46.271259 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:46.271609 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:46.272031 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:46.272034 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:46.272036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:46.272212 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:46.273025 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:46.273031 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:46.273306 (+ 275us) client_negotiation.cc:769] Sending connection context
0929 03:17:46.273500 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 03:17:46.273800 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":120,"threads_started":1}
I20260929 03:17:46.276432 27236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:46.258648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:45801 (local address 127.13.238.190:46833)
0929 03:17:46.259230 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:46.259249 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:46.259293 (+ 44us) server_negotiation.cc:407] Connection header received
0929 03:17:46.259586 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:46.259602 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:46.260021 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:46.260367 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:46.261997 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:46.263332 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:46.264855 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:46.265580 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:46.265739 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:46.268968 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:46.268975 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:46.268991 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:46.269026 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:46.271096 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:46.271754 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:46.271761 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:46.271767 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:46.271916 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:46.272329 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:46.272333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:46.272335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:46.272850 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:46.272971 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:46.275292 (+ 2321us) server_negotiation.cc:299] Negotiation successful
0929 03:17:46.275573 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":116,"threads_started":1}
I20260929 03:17:46.279250 27233 heartbeater.cc:347] Connected to a master server at 127.13.238.190:46833
I20260929 03:17:46.279475 27233 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:46.279937 27233 heartbeater.cc:511] Master 127.13.238.190:46833 requested a full tablet report, sending...
W20260929 03:17:46.282094 27233 heartbeater.cc:644] Failed to heartbeat to 127.13.238.190:46833 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 03:17:46.282261 27233 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:46.282452 27233 heartbeater.cc:511] Master 127.13.238.190:46833 requested a full tablet report, sending...
I20260929 03:17:46.283318 27233 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:46.283499 27233 heartbeater.cc:511] Master 127.13.238.190:46833 requested a full tablet report, sending...
W20260929 03:17:46.284264 27233 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:17:46.388903 27237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:46.389145 27237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:46.389212 27237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:46.393232 27237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:46.393287 27237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:46.393358 27237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:17:46.398736 27237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:35591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.238.148:52131/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:40965
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=43339
--tserver_master_addrs=127.13.238.190:46833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:17:46.399729 27237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:46.400701 27237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:46.409837 27245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:46.409955 27242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:46.409837 27243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:46.410645 27237 server_base.cc:1034] running on GCE node
I20260929 03:17:46.411038 27237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:46.411659 27237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:46.412824 27237 hybrid_clock.cc:648] HybridClock initialized: now 1790651866412798 us; error 36 us; skew 500 ppm
Sep 29 03:17:46 dist-test-slave-nbjg krb5kdc[26039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651866, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:46.415916 27237 init.cc:377] Logged in from keytab as kudu/127.13.238.130@KRBTEST.COM (short username kudu)
I20260929 03:17:46.417122 27237 webserver.cc:460] Webserver started at http://127.13.238.130:43339/ using document root <none> and password file <none>
I20260929 03:17:46.417779 27237 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:46.417873 27237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 03:17:46 dist-test-slave-nbjg krb5kdc[26039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651866, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for HTTP/127.13.238.148@KRBTEST.COM
I20260929 03:17:46.440456 27237 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 03:17:46.443512 27253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:46.445063 27237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:17:46.445191 27237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "5a65c74e75cf4b43af0ccde9d37b0fae"
format_stamp: "Formatted at 2026-09-29 03:17:45 on dist-test-slave-nbjg"
server_key: "768de0f69854101e0207959a5005a537"
server_key_iv: "576719b04b09dda698e12864aa1d2940"
server_key_version: "kuduclusterkey@0"
I20260929 03:17:46.445865 27237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:46.462936 27237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:46.466332 27237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:46.466517 27237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:46.467244 27237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:46.468271 27237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:46.468375 27237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:46.468469 27237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:46.468533 27237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:46.479292 27237 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:40965
I20260929 03:17:46.479344 27366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:40965 every 8 connection(s)
I20260929 03:17:46.480659 27237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:17:46.488129 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 27237
I20260929 03:17:46.488600 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.238.148:52131/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:36927
--local_ip_for_outbound_sockets=127.13.238.131
--tserver_master_addrs=127.13.238.190:46833
--webserver_port=45607
--webserver_interface=127.13.238.131
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:35591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.238.148:52131/kms with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:17:46 dist-test-slave-nbjg krb5kdc[26039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651866, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:17:46.495440 27236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:46.484466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:33469 (local address 127.13.238.190:46833)
0929 03:17:46.484613 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:46.484618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:46.484636 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:17:46.485257 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:46.485261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:46.485321 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:46.485377 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:46.486789 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:46.487313 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:46.488410 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:46.488650 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:46.488738 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:46.491367 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:46.491371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:46.491374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:46.491402 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:46.493001 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:46.493611 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:46.493615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:46.493617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:46.493702 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:46.494088 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:46.494092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:46.494106 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:46.494363 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:46.494440 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:46.495137 (+ 697us) server_negotiation.cc:299] Negotiation successful
0929 03:17:46.495269 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:17:46.497278 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:46.483497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:46833 (local address 127.13.238.130:33469)
0929 03:17:46.483991 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:46.484042 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:46.484981 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:46.485810 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:46.485819 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:46.486122 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:46.486641 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:46.486655 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:46.487429 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:46.487432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:46.488292 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:46.488301 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:46.488787 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:46.488791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:46.488989 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:46.489612 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:46.489634 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:46.491174 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:46.493141 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:46.493148 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:46.493163 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:46.493494 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:46.493809 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:46.493812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:46.493814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:46.493978 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:46.494520 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:46.494526 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:46.494834 (+ 308us) client_negotiation.cc:769] Sending connection context
0929 03:17:46.495876 (+ 1042us) client_negotiation.cc:240] Negotiation successful
0929 03:17:46.496223 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":113,"threads_started":1}
I20260929 03:17:46.498970 27367 heartbeater.cc:347] Connected to a master server at 127.13.238.190:46833
I20260929 03:17:46.499214 27367 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:46.499796 27367 heartbeater.cc:511] Master 127.13.238.190:46833 requested a full tablet report, sending...
W20260929 03:17:46.501509 27367 heartbeater.cc:644] Failed to heartbeat to 127.13.238.190:46833 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 03:17:46.501634 27367 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:46.501798 27367 heartbeater.cc:511] Master 127.13.238.190:46833 requested a full tablet report, sending...
I20260929 03:17:46.502684 27367 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:46.502847 27367 heartbeater.cc:511] Master 127.13.238.190:46833 requested a full tablet report, sending...
W20260929 03:17:46.503656 27367 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:17:46.622593 27370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:46.622833 27370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:46.622871 27370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:46.626967 27370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:46.627022 27370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:46.627077 27370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:17:46.633168 27370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:35591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.238.148:52131/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:36927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=45607
--tserver_master_addrs=127.13.238.190:46833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:17:46.634627 27370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:46.635973 27370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:46.648226 27377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:46.648823 27376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:46.649399 27379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:46.650311 27370 server_base.cc:1034] running on GCE node
I20260929 03:17:46.651072 27370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:46.651940 27370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:46.653154 27370 hybrid_clock.cc:648] HybridClock initialized: now 1790651866653121 us; error 46 us; skew 500 ppm
Sep 29 03:17:46 dist-test-slave-nbjg krb5kdc[26039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651866, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:46.656723 27370 init.cc:377] Logged in from keytab as kudu/127.13.238.131@KRBTEST.COM (short username kudu)
I20260929 03:17:46.657812 27370 webserver.cc:460] Webserver started at http://127.13.238.131:45607/ using document root <none> and password file <none>
I20260929 03:17:46.658550 27370 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:46.658653 27370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 03:17:46 dist-test-slave-nbjg krb5kdc[26039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651866, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for HTTP/127.13.238.148@KRBTEST.COM
I20260929 03:17:46.683526 27370 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:17:46.685957 27387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:46.687532 27370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:17:46.687668 27370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "444f2fdba3644bbb8c971717fa919ba9"
format_stamp: "Formatted at 2026-09-29 03:17:45 on dist-test-slave-nbjg"
server_key: "fcd71039da1c1f8e77e60072ce6db300"
server_key_iv: "6fa0e921539a47d7baabbc002ebaeeb6"
server_key_version: "kuduclusterkey@0"
I20260929 03:17:46.688400 27370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:46.729655 27370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:46.732714 27370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:46.732900 27370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:46.733489 27370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:46.734377 27370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:46.734428 27370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:46.734481 27370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:46.734515 27370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:46.744299 27370 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:36927
I20260929 03:17:46.744331 27500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:36927 every 8 connection(s)
I20260929 03:17:46.745342 27370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:17:46.748489 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 27370
Sep 29 03:17:46 dist-test-slave-nbjg krb5kdc[26039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651866, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:17:46.759555 27236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:46.747245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:38597 (local address 127.13.238.190:46833)
0929 03:17:46.747382 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:46.747386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:46.748339 (+ 953us) server_negotiation.cc:407] Connection header received
0929 03:17:46.749946 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:46.749950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:46.750012 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:46.750096 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:46.751215 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:46.751756 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:46.753153 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:46.753363 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:46.753443 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:46.755757 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:46.755760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:46.755762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:46.755783 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:46.757389 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:46.757882 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:46.757885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:46.757887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:46.757966 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:46.758388 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:46.758390 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:46.758391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:46.758551 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:46.758635 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:46.759247 (+ 612us) server_negotiation.cc:299] Negotiation successful
0929 03:17:46.759384 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":58}
I20260929 03:17:46.759908 27504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:46.749133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52732 (local address 127.13.238.190:46833)
0929 03:17:46.750085 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:46.750090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:46.750102 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:17:46.750139 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:46.750141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:46.750182 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:46.750302 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:46.751868 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:46.752602 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:46.753815 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:46.754066 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:46.754153 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:46.756008 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:46.756011 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:46.756012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:46.756046 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:46.757503 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:46.757943 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:46.757946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:46.757948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:46.758027 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:46.758630 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:46.758633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:46.758635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:46.758810 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:46.758995 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:46.759209 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 03:17:46.759772 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":885,"spinlock_wait_cycles":896,"thread_start_us":69,"threads_started":1}
I20260929 03:17:46.760730 27503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:46.747701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:46833 (local address 127.13.238.131:38597)
0929 03:17:46.748186 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:46.748239 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:46.749716 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:46.750327 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:46.750336 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:46.750615 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:46.751082 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:46.751096 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:46.751869 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:46.751872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:46.753051 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:46.753060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:46.753490 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:46.753503 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:46.753655 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:46.754289 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:46.754305 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:46.755607 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:46.757519 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:46.757524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:46.757540 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:46.757793 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:46.758071 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:46.758075 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:46.758077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:46.758296 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:46.758687 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:46.758693 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:46.758985 (+ 292us) client_negotiation.cc:769] Sending connection context
0929 03:17:46.759310 (+ 325us) client_negotiation.cc:240] Negotiation successful
0929 03:17:46.759610 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":108,"threads_started":1}
I20260929 03:17:46.762054 27501 heartbeater.cc:347] Connected to a master server at 127.13.238.190:46833
I20260929 03:17:46.762344 27501 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:46.762916 27501 heartbeater.cc:511] Master 127.13.238.190:46833 requested a full tablet report, sending...
W20260929 03:17:46.764588 27501 heartbeater.cc:644] Failed to heartbeat to 127.13.238.190:46833 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 03:17:46.764720 27501 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:46.764890 27501 heartbeater.cc:511] Master 127.13.238.190:46833 requested a full tablet report, sending...
I20260929 03:17:46.767066 27501 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:46.767227 27501 heartbeater.cc:511] Master 127.13.238.190:46833 requested a full tablet report, sending...
W20260929 03:17:46.767982 27501 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 03:17:47 dist-test-slave-nbjg krb5kdc[26039](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790651867, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:47.284529 27233 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:47.284899 27233 heartbeater.cc:511] Master 127.13.238.190:46833 requested a full tablet report, sending...
I20260929 03:17:47.286502 27035 ts_manager.cc:194] Registered new tserver with Master: 5ec3f493ee5345b9aac606ba82973364 (127.13.238.129:40163)
I20260929 03:17:47.287783 27035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.129@KRBTEST.COM'} at 127.13.238.129:45801
Sep 29 03:17:47 dist-test-slave-nbjg krb5kdc[26039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651867, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for HTTP/127.13.238.148@KRBTEST.COM
I20260929 03:17:47.503901 27367 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:47.504204 27367 heartbeater.cc:511] Master 127.13.238.190:46833 requested a full tablet report, sending...
I20260929 03:17:47.504930 27035 ts_manager.cc:194] Registered new tserver with Master: 5a65c74e75cf4b43af0ccde9d37b0fae (127.13.238.130:40965)
I20260929 03:17:47.505545 27035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.130@KRBTEST.COM'} at 127.13.238.130:33469
Sep 29 03:17:47 dist-test-slave-nbjg krb5kdc[26039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651867, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for HTTP/127.13.238.148@KRBTEST.COM
I20260929 03:17:47.768214 27501 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:47.768510 27501 heartbeater.cc:511] Master 127.13.238.190:46833 requested a full tablet report, sending...
I20260929 03:17:47.769243 27035 ts_manager.cc:194] Registered new tserver with Master: 444f2fdba3644bbb8c971717fa919ba9 (127.13.238.131:36927)
I20260929 03:17:47.769927 27035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.131@KRBTEST.COM'} at 127.13.238.131:38597
I20260929 03:17:47.778506 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:17:47.790707 27525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:47.780062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52744 (local address 127.13.238.190:46833)
0929 03:17:47.780814 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:47.780818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:47.780833 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:17:47.780870 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:47.780873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:47.780919 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:47.781012 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:47.782004 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:47.782944 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:47.784153 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:47.784529 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:47.784624 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:47.786452 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:47.786458 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:47.786461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:47.786488 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:47.788251 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:47.788753 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:47.788757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:47.788758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:47.788849 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:47.789148 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:47.789152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:47.789154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:47.789376 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:47.789497 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:47.790376 (+ 879us) server_negotiation.cc:299] Negotiation successful
0929 03:17:47.790508 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":77,"threads_started":1}
I20260929 03:17:47.795293 27035 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:17:47.800283 27035 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:17:47.829298 27530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:47.818806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:40965 (local address 127.0.0.1:45886)
0929 03:17:47.822780 (+ 3974us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:47.822821 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:47.822950 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:47.823692 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:47.823698 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:47.823723 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:47.823863 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:47.823872 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:47.825559 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:47.825562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:47.826989 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:47.826998 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:47.828758 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:47.828762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:47.828885 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:47.828921 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 03:17:47.829021 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 03:17:47.829137 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3814,"thread_start_us":129,"threads_started":1}
I20260929 03:17:47.829747 27531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:47.819568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45886 (local address 127.13.238.130:40965)
0929 03:17:47.822573 (+ 3005us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:47.822580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:47.823091 (+ 511us) server_negotiation.cc:407] Connection header received
0929 03:17:47.823177 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:47.823182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:47.823372 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:47.823512 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:47.823998 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:47.825435 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:47.827120 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:47.828425 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:47.828551 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:47.828697 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:47.829493 (+ 796us) server_negotiation.cc:299] Negotiation successful
0929 03:17:47.829605 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2890,"thread_start_us":78,"threads_started":1}
I20260929 03:17:47.833019 27297 tablet_service.cc:1467] Processing CreateTablet for tablet fa68958d06a74fe1af442e1be6251b43 (DEFAULT_TABLE table=test-table [id=00ef901733794202a2fbe0b306ce124c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:47.834219 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:47.820452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:40163 (local address 127.0.0.1:56878)
0929 03:17:47.826431 (+ 5979us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:47.826445 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:47.826524 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:47.827382 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:47.827385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:47.827404 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:47.827484 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:47.827490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:47.829239 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:47.829242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:47.832440 (+ 3198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:47.832450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:47.833793 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:47.833800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:47.833886 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:47.833909 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:17:47.833979 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 03:17:47.834046 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5909,"thread_start_us":95,"threads_started":1}
I20260929 03:17:47.834457 27297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa68958d06a74fe1af442e1be6251b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:47.834738 27533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:47.820619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56878 (local address 127.13.238.129:40163)
0929 03:17:47.826758 (+ 6139us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:47.826765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:47.826789 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:17:47.826860 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:47.826865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:47.827082 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:47.827234 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:47.827603 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:47.829127 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:47.832617 (+ 3490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:47.833470 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:47.833590 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:47.833725 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:47.834496 (+ 771us) server_negotiation.cc:299] Negotiation successful
0929 03:17:47.834599 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6029,"thread_start_us":112,"threads_started":1}
I20260929 03:17:47.837848 27160 tablet_service.cc:1467] Processing CreateTablet for tablet fa68958d06a74fe1af442e1be6251b43 (DEFAULT_TABLE table=test-table [id=00ef901733794202a2fbe0b306ce124c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:47.838938 27160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa68958d06a74fe1af442e1be6251b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:47.840175 27534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:47.821201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:36927 (local address 127.0.0.1:35760)
0929 03:17:47.827120 (+ 5919us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:47.827136 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:47.827236 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:47.828330 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:47.828333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:47.828353 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:47.828439 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:47.828445 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:47.837627 (+ 9182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:47.837630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:47.838728 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:47.838739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:47.839793 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:47.839797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:47.839894 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:47.839922 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 03:17:47.839972 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 03:17:47.840036 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5846,"thread_start_us":89,"threads_started":1}
I20260929 03:17:47.840785 27535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:47.821366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35760 (local address 127.13.238.131:36927)
0929 03:17:47.827579 (+ 6213us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:47.827586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:47.827623 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:17:47.827714 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:47.827720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:47.827987 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:47.828153 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:47.835817 (+ 7664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:47.837503 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:47.838870 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:47.839637 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:47.840370 (+ 733us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:47.840500 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:47.840547 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 03:17:47.840651 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6063,"thread_start_us":85,"threads_started":1}
I20260929 03:17:47.843724 27435 tablet_service.cc:1467] Processing CreateTablet for tablet fa68958d06a74fe1af442e1be6251b43 (DEFAULT_TABLE table=test-table [id=00ef901733794202a2fbe0b306ce124c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:47.844802 27435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa68958d06a74fe1af442e1be6251b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:47.847960 27536 tablet_bootstrap.cc:492] T fa68958d06a74fe1af442e1be6251b43 P 5a65c74e75cf4b43af0ccde9d37b0fae: Bootstrap starting.
I20260929 03:17:47.850466 27536 tablet_bootstrap.cc:654] T fa68958d06a74fe1af442e1be6251b43 P 5a65c74e75cf4b43af0ccde9d37b0fae: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:47.850947 27537 tablet_bootstrap.cc:492] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364: Bootstrap starting.
I20260929 03:17:47.852828 27536 log.cc:824] T fa68958d06a74fe1af442e1be6251b43 P 5a65c74e75cf4b43af0ccde9d37b0fae: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:47.853353 27538 tablet_bootstrap.cc:492] T fa68958d06a74fe1af442e1be6251b43 P 444f2fdba3644bbb8c971717fa919ba9: Bootstrap starting.
I20260929 03:17:47.854669 27537 tablet_bootstrap.cc:654] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:47.855469 27538 tablet_bootstrap.cc:654] T fa68958d06a74fe1af442e1be6251b43 P 444f2fdba3644bbb8c971717fa919ba9: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:47.855547 27537 log.cc:824] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:47.856314 27538 log.cc:824] T fa68958d06a74fe1af442e1be6251b43 P 444f2fdba3644bbb8c971717fa919ba9: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:47.857792 27536 tablet_bootstrap.cc:492] T fa68958d06a74fe1af442e1be6251b43 P 5a65c74e75cf4b43af0ccde9d37b0fae: No bootstrap required, opened a new log
I20260929 03:17:47.858716 27538 tablet_bootstrap.cc:492] T fa68958d06a74fe1af442e1be6251b43 P 444f2fdba3644bbb8c971717fa919ba9: No bootstrap required, opened a new log
I20260929 03:17:47.858965 27538 ts_tablet_manager.cc:1397] T fa68958d06a74fe1af442e1be6251b43 P 444f2fdba3644bbb8c971717fa919ba9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 03:17:47.859112 27537 tablet_bootstrap.cc:492] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364: No bootstrap required, opened a new log
I20260929 03:17:47.859360 27536 ts_tablet_manager.cc:1397] T fa68958d06a74fe1af442e1be6251b43 P 5a65c74e75cf4b43af0ccde9d37b0fae: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260929 03:17:47.862391 27537 ts_tablet_manager.cc:1397] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260929 03:17:47.863198 27538 raft_consensus.cc:348] T fa68958d06a74fe1af442e1be6251b43 P 444f2fdba3644bbb8c971717fa919ba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a65c74e75cf4b43af0ccde9d37b0fae" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 40965 } } peers { permanent_uuid: "5ec3f493ee5345b9aac606ba82973364" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 40163 } } peers { permanent_uuid: "444f2fdba3644bbb8c971717fa919ba9" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36927 } }
I20260929 03:17:47.863510 27538 raft_consensus.cc:374] T fa68958d06a74fe1af442e1be6251b43 P 444f2fdba3644bbb8c971717fa919ba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:47.863579 27538 raft_consensus.cc:729] T fa68958d06a74fe1af442e1be6251b43 P 444f2fdba3644bbb8c971717fa919ba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 444f2fdba3644bbb8c971717fa919ba9, State: Initialized, Role: FOLLOWER
I20260929 03:17:47.864122 27538 consensus_queue.cc:260] T fa68958d06a74fe1af442e1be6251b43 P 444f2fdba3644bbb8c971717fa919ba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a65c74e75cf4b43af0ccde9d37b0fae" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 40965 } } peers { permanent_uuid: "5ec3f493ee5345b9aac606ba82973364" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 40163 } } peers { permanent_uuid: "444f2fdba3644bbb8c971717fa919ba9" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36927 } }
I20260929 03:17:47.865226 27538 ts_tablet_manager.cc:1428] T fa68958d06a74fe1af442e1be6251b43 P 444f2fdba3644bbb8c971717fa919ba9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 03:17:47.865630 27501 heartbeater.cc:503] Master 127.13.238.190:46833 was elected leader, sending a full tablet report...
I20260929 03:17:47.865599 27536 raft_consensus.cc:348] T fa68958d06a74fe1af442e1be6251b43 P 5a65c74e75cf4b43af0ccde9d37b0fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a65c74e75cf4b43af0ccde9d37b0fae" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 40965 } } peers { permanent_uuid: "5ec3f493ee5345b9aac606ba82973364" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 40163 } } peers { permanent_uuid: "444f2fdba3644bbb8c971717fa919ba9" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36927 } }
I20260929 03:17:47.865854 27536 raft_consensus.cc:374] T fa68958d06a74fe1af442e1be6251b43 P 5a65c74e75cf4b43af0ccde9d37b0fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:47.865901 27536 raft_consensus.cc:729] T fa68958d06a74fe1af442e1be6251b43 P 5a65c74e75cf4b43af0ccde9d37b0fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a65c74e75cf4b43af0ccde9d37b0fae, State: Initialized, Role: FOLLOWER
I20260929 03:17:47.868580 27537 raft_consensus.cc:348] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a65c74e75cf4b43af0ccde9d37b0fae" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 40965 } } peers { permanent_uuid: "5ec3f493ee5345b9aac606ba82973364" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 40163 } } peers { permanent_uuid: "444f2fdba3644bbb8c971717fa919ba9" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36927 } }
I20260929 03:17:47.869114 27536 consensus_queue.cc:260] T fa68958d06a74fe1af442e1be6251b43 P 5a65c74e75cf4b43af0ccde9d37b0fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a65c74e75cf4b43af0ccde9d37b0fae" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 40965 } } peers { permanent_uuid: "5ec3f493ee5345b9aac606ba82973364" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 40163 } } peers { permanent_uuid: "444f2fdba3644bbb8c971717fa919ba9" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36927 } }
I20260929 03:17:47.870285 27536 ts_tablet_manager.cc:1428] T fa68958d06a74fe1af442e1be6251b43 P 5a65c74e75cf4b43af0ccde9d37b0fae: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260929 03:17:47.870293 27537 raft_consensus.cc:374] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:47.870349 27537 raft_consensus.cc:729] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec3f493ee5345b9aac606ba82973364, State: Initialized, Role: FOLLOWER
I20260929 03:17:47.870821 27367 heartbeater.cc:503] Master 127.13.238.190:46833 was elected leader, sending a full tablet report...
I20260929 03:17:47.870798 27537 consensus_queue.cc:260] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a65c74e75cf4b43af0ccde9d37b0fae" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 40965 } } peers { permanent_uuid: "5ec3f493ee5345b9aac606ba82973364" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 40163 } } peers { permanent_uuid: "444f2fdba3644bbb8c971717fa919ba9" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36927 } }
I20260929 03:17:47.871786 27537 ts_tablet_manager.cc:1428] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364: Time spent starting tablet: real 0.009s user 0.002s sys 0.003s
I20260929 03:17:47.872118 27233 heartbeater.cc:503] Master 127.13.238.190:46833 was elected leader, sending a full tablet report...
W20260929 03:17:47.982898 27368 tablet.cc:2369] T fa68958d06a74fe1af442e1be6251b43 P 5a65c74e75cf4b43af0ccde9d37b0fae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:17:47.997511 27502 tablet.cc:2369] T fa68958d06a74fe1af442e1be6251b43 P 444f2fdba3644bbb8c971717fa919ba9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:17:48.007416 27234 tablet.cc:2369] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:17:48.163744 27544 raft_consensus.cc:482] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:17:48.163976 27544 raft_consensus.cc:504] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a65c74e75cf4b43af0ccde9d37b0fae" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 40965 } } peers { permanent_uuid: "5ec3f493ee5345b9aac606ba82973364" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 40163 } } peers { permanent_uuid: "444f2fdba3644bbb8c971717fa919ba9" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36927 } }
I20260929 03:17:48.165194 27544 leader_election.cc:288] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a65c74e75cf4b43af0ccde9d37b0fae (127.13.238.130:40965), 444f2fdba3644bbb8c971717fa919ba9 (127.13.238.131:36927)
I20260929 03:17:48.169013 27531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:48.165592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:49175 (local address 127.13.238.130:40965)
0929 03:17:48.165736 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:48.165741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:48.166016 (+ 275us) server_negotiation.cc:407] Connection header received
0929 03:17:48.166135 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:48.166138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:48.166181 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:48.166270 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:48.166557 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:48.167063 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:48.168119 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:48.168505 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:48.168609 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:48.168673 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:48.168820 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 03:17:48.168888 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:17:48.169032 27552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:48.165451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:40965 (local address 127.13.238.129:49175)
0929 03:17:48.165925 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:48.165936 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:48.166035 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:48.166279 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:48.166282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:48.166309 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:48.166458 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:48.166464 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:48.167168 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:48.167171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:48.167996 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:48.168005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:48.168609 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:48.168613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:48.168708 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:48.168731 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:17:48.168797 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 03:17:48.168868 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":46,"thread_start_us":114,"threads_started":1}
I20260929 03:17:48.169225 27535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:48.165585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:51347 (local address 127.13.238.131:36927)
0929 03:17:48.165723 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:48.165729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:48.165921 (+ 192us) server_negotiation.cc:407] Connection header received
0929 03:17:48.166066 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:48.166068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:48.166110 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:48.166173 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:48.166632 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:48.167143 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:48.168340 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:48.168740 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:48.168847 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:48.168896 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:48.169051 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 03:17:48.169121 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:17:48.169255 27553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:48.165453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:36927 (local address 127.13.238.129:51347)
0929 03:17:48.165834 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:48.165846 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:48.165956 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:48.166223 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:48.166227 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:48.166242 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:48.166476 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:48.166483 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:48.167250 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:48.167254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:48.168210 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:48.168220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:48.168849 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:48.168853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:48.168945 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:48.168964 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 03:17:48.169035 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 03:17:48.169100 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":2,"spinlock_wait_cycles":38784,"thread_start_us":119,"threads_started":1}
I20260929 03:17:48.169780 27303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa68958d06a74fe1af442e1be6251b43" candidate_uuid: "5ec3f493ee5345b9aac606ba82973364" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a65c74e75cf4b43af0ccde9d37b0fae" is_pre_election: true
I20260929 03:17:48.169803 27455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa68958d06a74fe1af442e1be6251b43" candidate_uuid: "5ec3f493ee5345b9aac606ba82973364" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "444f2fdba3644bbb8c971717fa919ba9" is_pre_election: true
I20260929 03:17:48.170056 27455 raft_consensus.cc:2454] T fa68958d06a74fe1af442e1be6251b43 P 444f2fdba3644bbb8c971717fa919ba9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ec3f493ee5345b9aac606ba82973364 in term 0.
I20260929 03:17:48.170070 27303 raft_consensus.cc:2454] T fa68958d06a74fe1af442e1be6251b43 P 5a65c74e75cf4b43af0ccde9d37b0fae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ec3f493ee5345b9aac606ba82973364 in term 0.
I20260929 03:17:48.170547 27121 leader_election.cc:302] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 444f2fdba3644bbb8c971717fa919ba9, 5ec3f493ee5345b9aac606ba82973364; no voters:
I20260929 03:17:48.170797 27544 raft_consensus.cc:2781] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:17:48.170884 27544 raft_consensus.cc:482] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:17:48.170917 27544 raft_consensus.cc:3037] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:48.171944 27544 raft_consensus.cc:504] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a65c74e75cf4b43af0ccde9d37b0fae" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 40965 } } peers { permanent_uuid: "5ec3f493ee5345b9aac606ba82973364" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 40163 } } peers { permanent_uuid: "444f2fdba3644bbb8c971717fa919ba9" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36927 } }
I20260929 03:17:48.172355 27544 leader_election.cc:288] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364 [CANDIDATE]: Term 1 election: Requested vote from peers 5a65c74e75cf4b43af0ccde9d37b0fae (127.13.238.130:40965), 444f2fdba3644bbb8c971717fa919ba9 (127.13.238.131:36927)
I20260929 03:17:48.172698 27303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa68958d06a74fe1af442e1be6251b43" candidate_uuid: "5ec3f493ee5345b9aac606ba82973364" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a65c74e75cf4b43af0ccde9d37b0fae"
I20260929 03:17:48.172706 27455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa68958d06a74fe1af442e1be6251b43" candidate_uuid: "5ec3f493ee5345b9aac606ba82973364" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "444f2fdba3644bbb8c971717fa919ba9"
I20260929 03:17:48.172811 27455 raft_consensus.cc:3037] T fa68958d06a74fe1af442e1be6251b43 P 444f2fdba3644bbb8c971717fa919ba9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:48.172843 27303 raft_consensus.cc:3037] T fa68958d06a74fe1af442e1be6251b43 P 5a65c74e75cf4b43af0ccde9d37b0fae [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:48.173772 27455 raft_consensus.cc:2454] T fa68958d06a74fe1af442e1be6251b43 P 444f2fdba3644bbb8c971717fa919ba9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ec3f493ee5345b9aac606ba82973364 in term 1.
I20260929 03:17:48.173848 27303 raft_consensus.cc:2454] T fa68958d06a74fe1af442e1be6251b43 P 5a65c74e75cf4b43af0ccde9d37b0fae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ec3f493ee5345b9aac606ba82973364 in term 1.
I20260929 03:17:48.174036 27121 leader_election.cc:302] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 444f2fdba3644bbb8c971717fa919ba9, 5ec3f493ee5345b9aac606ba82973364; no voters:
I20260929 03:17:48.174209 27544 raft_consensus.cc:2781] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:48.174444 27544 raft_consensus.cc:686] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364 [term 1 LEADER]: Becoming Leader. State: Replica: 5ec3f493ee5345b9aac606ba82973364, State: Running, Role: LEADER
I20260929 03:17:48.174785 27544 consensus_queue.cc:237] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a65c74e75cf4b43af0ccde9d37b0fae" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 40965 } } peers { permanent_uuid: "5ec3f493ee5345b9aac606ba82973364" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 40163 } } peers { permanent_uuid: "444f2fdba3644bbb8c971717fa919ba9" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36927 } }
I20260929 03:17:48.177892 27035 catalog_manager.cc:5495] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ec3f493ee5345b9aac606ba82973364 (127.13.238.129). New cstate: current_term: 1 leader_uuid: "5ec3f493ee5345b9aac606ba82973364" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a65c74e75cf4b43af0ccde9d37b0fae" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 40965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ec3f493ee5345b9aac606ba82973364" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 40163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "444f2fdba3644bbb8c971717fa919ba9" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36927 } health_report { overall_health: UNKNOWN } } }
I20260929 03:17:48.189292 27533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:48.185798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56882 (local address 127.13.238.129:40163)
0929 03:17:48.185919 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:48.185922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:48.185936 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:17:48.186003 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:48.186005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:48.186056 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:48.186153 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:17:48.186445 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:48.186986 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:48.187879 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:48.188143 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:48.188226 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:48.188460 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:17:48.188891 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:17:48.188981 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:48.189103 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 03:17:48.189157 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 03:17:48.195300 27303 raft_consensus.cc:1261] T fa68958d06a74fe1af442e1be6251b43 P 5a65c74e75cf4b43af0ccde9d37b0fae [term 1 FOLLOWER]: Refusing update from remote peer 5ec3f493ee5345b9aac606ba82973364: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:48.195297 27455 raft_consensus.cc:1261] T fa68958d06a74fe1af442e1be6251b43 P 444f2fdba3644bbb8c971717fa919ba9 [term 1 FOLLOWER]: Refusing update from remote peer 5ec3f493ee5345b9aac606ba82973364: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:48.195987 27554 consensus_queue.cc:1035] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a65c74e75cf4b43af0ccde9d37b0fae" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 40965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:48.196163 27544 consensus_queue.cc:1035] T fa68958d06a74fe1af442e1be6251b43 P 5ec3f493ee5345b9aac606ba82973364 [LEADER]: Connected to new peer: Peer: permanent_uuid: "444f2fdba3644bbb8c971717fa919ba9" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:48.202335 27557 mvcc.cc:205] Tried to move back new op lower bound from 7334510052120436736 to 7334510052046499840. Current Snapshot: MvccSnapshot[applied={T|T < 7334510052120436736}]
I20260929 03:17:48.205003 27559 mvcc.cc:205] Tried to move back new op lower bound from 7334510052120436736 to 7334510052046499840. Current Snapshot: MvccSnapshot[applied={T|T < 7334510052120436736}]
I20260929 03:17:48.211491 27030 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:17:48.211649 27030 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:17:48.213922 27030 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c4242b6615d44b7bd49150ab213ac10: Sending DeleteTablet for 3 replicas of tablet fa68958d06a74fe1af442e1be6251b43
I20260929 03:17:48.214689 27297 tablet_service.cc:1514] Processing DeleteTablet for tablet fa68958d06a74fe1af442e1be6251b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:48 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:45886
I20260929 03:17:48.214720 27435 tablet_service.cc:1514] Processing DeleteTablet for tablet fa68958d06a74fe1af442e1be6251b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:48 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:35760
I20260929 03:17:48.215231 27566 tablet_replica.cc:330] T fa68958d06a74fe1af442e1be6251b43 P 444f2fdba3644bbb8c971717fa919ba9: stopping tablet replica
I20260929 03:17:48.215370 27160 tablet_service.cc:1514] Processing DeleteTablet for tablet fa68958d06a74fe1af442e1be6251b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:48 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:56878
I20260929 03:17:48.215566 27567 tablet_replica.cc:330] T fa68958d06a74fe1af442e1be6251b43 P 5a65c74e75cf4b43af0ccde9d37b0fae: stopping tablet replica
I20260929 03:17:48.215600 27566 raft_consensus.cc:2229] T fa68958d06a74fe1af442e1be6251b43 P 444f2fdba3644bbb8c971717fa919ba9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:17:48.215873 27567 raft_consensus.cc:2229] T fa68958d06a74fe1af442e1be6251b43 P 5a65c74e75cf4b43af0ccde9d37b0fae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:17:48.215889 27566 raft_consensus.cc:2258] T fa68958d06a74fe1af442e1be6251b43 P 444f2fdba3644bbb8c971717fa919ba9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:48.216289 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 27073
I20260929 03:17:48.216384 27567 raft_consensus.cc:2258] T fa68958d06a74fe1af442e1be6251b43 P 5a65c74e75cf4b43af0ccde9d37b0fae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:48.219076 27567 ts_tablet_manager.cc:1905] T fa68958d06a74fe1af442e1be6251b43 P 5a65c74e75cf4b43af0ccde9d37b0fae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:48.219076 27566 ts_tablet_manager.cc:1905] T fa68958d06a74fe1af442e1be6251b43 P 444f2fdba3644bbb8c971717fa919ba9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:48.221771 27567 ts_tablet_manager.cc:1918] T fa68958d06a74fe1af442e1be6251b43 P 5a65c74e75cf4b43af0ccde9d37b0fae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:17:48.221771 27566 ts_tablet_manager.cc:1918] T fa68958d06a74fe1af442e1be6251b43 P 444f2fdba3644bbb8c971717fa919ba9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:17:48.221866 27566 log.cc:1196] T fa68958d06a74fe1af442e1be6251b43 P 444f2fdba3644bbb8c971717fa919ba9: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/fa68958d06a74fe1af442e1be6251b43
I20260929 03:17:48.221894 27567 log.cc:1196] T fa68958d06a74fe1af442e1be6251b43 P 5a65c74e75cf4b43af0ccde9d37b0fae: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790651713532531-14266-0/minicluster-data/ts-1/wal/wals/fa68958d06a74fe1af442e1be6251b43
W20260929 03:17:48.222134 27019 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.238.129:40163 (error 108)
I20260929 03:17:48.222370 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 27237
I20260929 03:17:48.222390 27566 ts_tablet_manager.cc:1939] T fa68958d06a74fe1af442e1be6251b43 P 444f2fdba3644bbb8c971717fa919ba9: Deleting consensus metadata
I20260929 03:17:48.223915 27020 catalog_manager.cc:4841] TS 444f2fdba3644bbb8c971717fa919ba9 (127.13.238.131:36927): tablet fa68958d06a74fe1af442e1be6251b43 (table test-table [id=00ef901733794202a2fbe0b306ce124c]) successfully deleted
I20260929 03:17:48.224491 27534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:48.224205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:40163 (local address 127.0.0.1:56892)
0929 03:17:48.224338 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:48.224410 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.129:40163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 03:17:48.224687 27019 catalog_manager.cc:4584] TS 5ec3f493ee5345b9aac606ba82973364 (127.13.238.129:40163): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fa68958d06a74fe1af442e1be6251b43: Network error: Client connection negotiation failed: client connection to 127.13.238.129:40163: connect: Connection refused (error 111)
I20260929 03:17:48.228081 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 27370
I20260929 03:17:48.228504 27534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:48.228250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:40965 (local address 127.0.0.1:45890)
0929 03:17:48.228390 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:48.228433 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.130:40965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 03:17:48.232802 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 27002
I20260929 03:17:48.240276 14266 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 03:17:48.574769 14266 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 03:17:48.574877 14266 mini_ranger.cc:66] Stopping Ranger...
I20260929 03:17:48.909319 14266 mini_ranger.cc:68] Stopped Ranger
2026-09-29 03:17:48.909 UTC [26069] LOG: received smart shutdown request
2026-09-29 03:17:48.916 UTC [26069] LOG: background worker "logical replication launcher" (PID 26078) exited with exit code 1
2026-09-29 03:17:48.916 UTC [26073] LOG: shutting down
2026-09-29 03:17:49.086 UTC [26069] LOG: database system is shut down
2026-09-29T03:17:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47576 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:17:49 dist-test-slave-nbjg krb5kdc[27578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:17:49 dist-test-slave-nbjg krb5kdc[27578](info): set up 2 sockets
Sep 29 03:17:49 dist-test-slave-nbjg krb5kdc[27578](info): commencing operation
krb5kdc: starting...
W20260929 03:17:50.150315 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:17:50 dist-test-slave-nbjg krb5kdc[27578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:17:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:17:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:17:50.281503 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:37329
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:35257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:37329,127.13.238.189:46779,127.13.238.188:33479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:50.389964 27591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:50.390280 27591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:50.390337 27591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:50.394646 27591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:50.394721 27591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:50.394752 27591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:50.394773 27591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:50.394793 27591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:17:50.400280 27591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:35257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:37329,127.13.238.189:46779,127.13.238.188:33479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:37329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:17:50.401378 27591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:50.402448 27591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:50.410703 27596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:50.410727 27597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:50.410728 27599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:50.410851 27591 server_base.cc:1034] running on GCE node
I20260929 03:17:50.411540 27591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:50.412654 27591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:50.413827 27591 hybrid_clock.cc:648] HybridClock initialized: now 1790651870413806 us; error 33 us; skew 500 ppm
Sep 29 03:17:50 dist-test-slave-nbjg krb5kdc[27578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651870, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:50.416601 27591 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:17:50.417569 27591 webserver.cc:460] Webserver started at http://127.13.238.190:46875/ using document root <none> and password file <none>
I20260929 03:17:50.418103 27591 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:50.418167 27591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:50.418388 27591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:50.420425 27591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "86b4f654fd5c4b999bfca6a69c875a14"
format_stamp: "Formatted at 2026-09-29 03:17:50 on dist-test-slave-nbjg"
server_key: "cce8550f71d3f482c3fcad7911f25461"
server_key_iv: "e8773de73dfd7dae07ba3dd1976e352b"
server_key_version: "encryptionkey@0"
I20260929 03:17:50.420961 27591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "86b4f654fd5c4b999bfca6a69c875a14"
format_stamp: "Formatted at 2026-09-29 03:17:50 on dist-test-slave-nbjg"
server_key: "cce8550f71d3f482c3fcad7911f25461"
server_key_iv: "e8773de73dfd7dae07ba3dd1976e352b"
server_key_version: "encryptionkey@0"
I20260929 03:17:50.424623 27591 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 03:17:50.427022 27606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:50.428215 27591 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260929 03:17:50.428323 27591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "86b4f654fd5c4b999bfca6a69c875a14"
format_stamp: "Formatted at 2026-09-29 03:17:50 on dist-test-slave-nbjg"
server_key: "cce8550f71d3f482c3fcad7911f25461"
server_key_iv: "e8773de73dfd7dae07ba3dd1976e352b"
server_key_version: "encryptionkey@0"
I20260929 03:17:50.428417 27591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:50.445875 27591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:50.449014 27591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:50.449186 27591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:50.455904 27591 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:37329
I20260929 03:17:50.455927 27658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:37329 every 8 connection(s)
I20260929 03:17:50.456830 27591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:17:50.457657 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 27591
I20260929 03:17:50.457779 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:17:50.458107 14266 external_mini_cluster.cc:1428] Setting key e6c27f255bf9dea8e9d687533bd87e4b
I20260929 03:17:50.460700 27659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:50.464416 27659 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37329 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.238.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.189@KRBTEST.COM" created.
Sep 29 03:17:50 dist-test-slave-nbjg krb5kdc[27578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651870, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:17:50.479844 27663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.466683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:37329 (local address 127.0.0.1:52156)
0929 03:17:50.467153 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.467201 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:50.468050 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:50.468820 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:50.468827 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:50.469260 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:50.469617 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:50.469635 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.470954 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.470959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:50.472118 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:50.472131 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.472762 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.472766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:50.472890 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.473435 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:50.473455 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:50.475104 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:50.476817 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:50.476825 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:50.476835 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:50.477147 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:50.477446 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:50.477449 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:50.477450 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:50.477563 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:50.478318 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:50.478324 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:50.478452 (+ 128us) client_negotiation.cc:769] Sending connection context
0929 03:17:50.478631 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 03:17:50.478996 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":86,"spinlock_wait_cycles":27008,"thread_start_us":148,"threads_started":1}
I20260929 03:17:50.479848 27662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.466683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52156 (local address 127.13.238.190:37329)
0929 03:17:50.467102 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:50.467117 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:50.467308 (+ 191us) server_negotiation.cc:407] Connection header received
0929 03:17:50.468377 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:50.468383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:50.468615 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:50.468818 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:50.469792 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.470812 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.472253 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.472613 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.472727 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.475238 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:50.475244 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:50.475255 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:50.475275 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:50.476682 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.477266 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.477271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.477272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.477376 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.477671 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.477675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.477677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.478061 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:50.478163 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:50.478732 (+ 569us) server_negotiation.cc:299] Negotiation successful
0929 03:17:50.478921 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":272,"thread_start_us":122,"threads_started":1}
I20260929 03:17:50.482178 27659 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46779 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.189.keytab.
Entry for principal kudu/127.13.238.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.189.keytab.
I20260929 03:17:50.482869 27663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.482628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:46779 (local address 127.0.0.1:43640)
0929 03:17:50.482746 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.482814 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.189:46779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 03:17:50.483086 27608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.238.189:46779: connect: Connection refused (error 111)
I20260929 03:17:50.483340 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.189:46779
--webserver_interface=127.13.238.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.189.keytab
--principal=kudu/127.13.238.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:35257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:37329,127.13.238.189:46779,127.13.238.188:33479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:17:50.483865 27663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.483699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:46779 (local address 127.0.0.1:43652)
0929 03:17:50.483775 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.483817 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.189:46779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 03:17:50.483990 27659 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.189:46779: Network error: Client connection negotiation failed: client connection to 127.13.238.189:46779: connect: Connection refused (error 111)
I20260929 03:17:50.533223 27659 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46779 } attempt: 1
I20260929 03:17:50.534102 27663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.533762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:46779 (local address 127.0.0.1:43666)
0929 03:17:50.533927 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.534000 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.189:46779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 03:17:50.534793 27663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.534583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:46779 (local address 127.0.0.1:43682)
0929 03:17:50.534677 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.534727 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.189:46779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 03:17:50.534912 27659 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.189:46779: Network error: Client connection negotiation failed: client connection to 127.13.238.189:46779: connect: Connection refused (error 111)
W20260929 03:17:50.591178 27666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:50.591440 27666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:50.591480 27666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:50.595454 27666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:50.595515 27666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:50.595532 27666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:50.595548 27666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:50.595568 27666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:17:50.600744 27666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:35257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:37329,127.13.238.189:46779,127.13.238.188:33479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.189.keytab
--principal=kudu/127.13.238.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.189:46779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.238.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:17:50.601763 27666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:50.602799 27666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:17:50.603075 27659 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46779 } attempt: 2
I20260929 03:17:50.603901 27663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.603573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:46779 (local address 127.0.0.1:43696)
0929 03:17:50.603739 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.603811 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.189:46779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 03:17:50.604580 27663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.604345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:46779 (local address 127.0.0.1:43708)
0929 03:17:50.604448 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.604511 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.189:46779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 03:17:50.604714 27659 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.189:46779: Network error: Client connection negotiation failed: client connection to 127.13.238.189:46779: connect: Connection refused (error 111)
W20260929 03:17:50.611394 27672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:50.611394 27674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:50.611560 27666 server_base.cc:1034] running on GCE node
W20260929 03:17:50.611394 27671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:50.612318 27666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:50.613427 27666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:50.614588 27666 hybrid_clock.cc:648] HybridClock initialized: now 1790651870614567 us; error 31 us; skew 500 ppm
Sep 29 03:17:50 dist-test-slave-nbjg krb5kdc[27578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651870, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:50.617524 27666 init.cc:377] Logged in from keytab as kudu/127.13.238.189@KRBTEST.COM (short username kudu)
I20260929 03:17:50.618515 27666 webserver.cc:460] Webserver started at http://127.13.238.189:42553/ using document root <none> and password file <none>
I20260929 03:17:50.619078 27666 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:50.619148 27666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:50.619364 27666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:50.621417 27666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-1/data/instance:
uuid: "3c2c6f019b9a4ae793595e8bab202eeb"
format_stamp: "Formatted at 2026-09-29 03:17:50 on dist-test-slave-nbjg"
server_key: "f1843dc527f3582aaccc1cea0d0f2dc1"
server_key_iv: "7a001b7d988f22a4cc03fbb8a6fe3b6d"
server_key_version: "encryptionkey@0"
I20260929 03:17:50.621969 27666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-1/wal/instance:
uuid: "3c2c6f019b9a4ae793595e8bab202eeb"
format_stamp: "Formatted at 2026-09-29 03:17:50 on dist-test-slave-nbjg"
server_key: "f1843dc527f3582aaccc1cea0d0f2dc1"
server_key_iv: "7a001b7d988f22a4cc03fbb8a6fe3b6d"
server_key_version: "encryptionkey@0"
I20260929 03:17:50.625640 27666 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:17:50.627995 27681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:50.629230 27666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 03:17:50.629339 27666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-1/wal
uuid: "3c2c6f019b9a4ae793595e8bab202eeb"
format_stamp: "Formatted at 2026-09-29 03:17:50 on dist-test-slave-nbjg"
server_key: "f1843dc527f3582aaccc1cea0d0f2dc1"
server_key_iv: "7a001b7d988f22a4cc03fbb8a6fe3b6d"
server_key_version: "encryptionkey@0"
I20260929 03:17:50.629431 27666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:50.656728 27666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:50.659883 27666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:50.660073 27666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:50.666761 27666 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.189:46779
I20260929 03:17:50.667006 27733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.189:46779 every 8 connection(s)
I20260929 03:17:50.667699 27666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-1/data/info.pb
I20260929 03:17:50.669224 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 27666
I20260929 03:17:50.669366 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-1/wal/instance
I20260929 03:17:50.669638 14266 external_mini_cluster.cc:1428] Setting key dbae17ef0dd9720086e636c0272507eb
I20260929 03:17:50.670482 27734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:50.674154 27734 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37329 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.238.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.188@KRBTEST.COM" created.
Sep 29 03:17:50 dist-test-slave-nbjg krb5kdc[27578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651870, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.189@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:17:50.687628 27662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.676119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52164 (local address 127.13.238.190:37329)
0929 03:17:50.676365 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:50.676369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:50.677309 (+ 940us) server_negotiation.cc:407] Connection header received
0929 03:17:50.678272 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:50.678275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:50.678338 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:50.678409 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:50.679652 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.680186 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.681447 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.681723 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.681814 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.684318 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:50.684321 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:50.684324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:50.684350 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:50.685660 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.686140 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.686143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.686144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.686251 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.686558 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.686561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.686563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.686747 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:50.686831 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:50.687337 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 03:17:50.687462 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260929 03:17:50.688481 27737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.676637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:37329 (local address 127.0.0.1:52164)
0929 03:17:50.677158 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.677201 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:50.678048 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:50.678593 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:50.678601 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:50.678937 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:50.679485 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:50.679506 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.680317 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.680321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:50.681312 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:50.681321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.681872 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.681875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:50.682073 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.682652 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:50.682667 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:50.684088 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:50.685771 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:50.685777 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:50.685794 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:50.686047 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:50.686332 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:50.686335 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:50.686337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:50.686471 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:50.686845 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:50.686851 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:50.687102 (+ 251us) client_negotiation.cc:769] Sending connection context
0929 03:17:50.687266 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 03:17:50.687597 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":109,"threads_started":1}
I20260929 03:17:50.689908 27734 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46779 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:17:50 dist-test-slave-nbjg krb5kdc[27578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651870, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.189@KRBTEST.COM for kudu/127.13.238.189@KRBTEST.COM
I20260929 03:17:50.695875 27659 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46779 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.188.keytab.
Entry for principal kudu/127.13.238.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.188.keytab.
I20260929 03:17:50.697228 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.188:33479
--webserver_interface=127.13.238.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.188.keytab
--principal=kudu/127.13.238.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:35257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:37329,127.13.238.189:46779,127.13.238.188:33479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:17:50.699832 27737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.690602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:46779 (local address 127.0.0.1:43718)
0929 03:17:50.690770 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.690784 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:50.690854 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:50.691425 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:50.691431 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:50.691613 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:50.691732 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:50.691740 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.693145 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.693148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:50.693720 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:50.693728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.694401 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.694404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:50.694475 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.694870 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:50.694883 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:50.696009 (+ 1126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:50.698243 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:50.698247 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:50.698249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:50.698488 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:50.698759 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:50.698763 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:50.698765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:50.698853 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:50.699355 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:50.699358 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:50.699450 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 03:17:50.699550 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 03:17:50.699669 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 03:17:50.699997 27739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.690690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43718 (local address 127.13.238.189:46779)
0929 03:17:50.690990 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:50.691001 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:50.691023 (+ 22us) server_negotiation.cc:407] Connection header received
0929 03:17:50.691073 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:50.691079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:50.691277 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:50.691411 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:50.691856 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.693014 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.693849 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.694298 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.694408 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.696176 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:50.696186 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:50.696200 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:50.696224 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:50.698113 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.698583 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.698589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.698591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.698669 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.698949 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.698953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.698955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.699255 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:50.699383 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:50.699642 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 03:17:50.699787 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":121,"threads_started":1}
Sep 29 03:17:50 dist-test-slave-nbjg krb5kdc[27578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651870, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for kudu/127.13.238.189@KRBTEST.COM
I20260929 03:17:50.702653 27734 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:17:50.703373 27737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.703064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:33479 (local address 127.0.0.1:46180)
0929 03:17:50.703234 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.703307 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:33479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260929 03:17:50.703534 27683 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.238.188:33479: connect: Connection refused (error 111)
I20260929 03:17:50.704487 27743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.704137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:33479 (local address 127.0.0.1:46194)
0929 03:17:50.704346 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.704395 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:33479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":80,"threads_started":1}
W20260929 03:17:50.704625 27734 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.188:33479: Network error: Client connection negotiation failed: client connection to 127.13.238.188:33479: connect: Connection refused (error 111)
I20260929 03:17:50.705586 27663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.696300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:46779 (local address 127.0.0.1:43728)
0929 03:17:50.696478 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.696492 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:50.696566 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:50.697043 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:50.697046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:50.697431 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:50.697562 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:50.697569 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.698385 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.698388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:50.699626 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:50.699637 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.700270 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.700274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:50.700389 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.700762 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:50.700776 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:50.702114 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:50.704082 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:50.704086 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:50.704088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:50.704338 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:50.704631 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:50.704634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:50.704637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:50.704732 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:50.705099 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:50.705102 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:50.705194 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 03:17:50.705278 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 03:17:50.705413 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260929 03:17:50.705615 27740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.696356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43728 (local address 127.13.238.189:46779)
0929 03:17:50.696833 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:50.696842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:50.696855 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:17:50.696882 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:50.696884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:50.696917 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:50.697012 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:50.697667 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.698176 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.699750 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.700068 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.700157 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.702282 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:50.702285 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:50.702287 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:50.702307 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:50.703950 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.704424 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.704427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.704429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.704505 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.704833 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.704836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.704838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.705004 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:50.705084 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:50.705337 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 03:17:50.705435 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":96,"threads_started":1}
I20260929 03:17:50.706532 27659 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:17:50.707212 27663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.706951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:33479 (local address 127.0.0.1:46198)
0929 03:17:50.707112 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.707156 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:33479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 03:17:50.707845 27663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.707627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:33479 (local address 127.0.0.1:46212)
0929 03:17:50.707754 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.707797 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:33479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260929 03:17:50.707944 27659 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.188:33479: Network error: Client connection negotiation failed: client connection to 127.13.238.188:33479: connect: Connection refused (error 111)
I20260929 03:17:50.739082 27659 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } attempt: 1
I20260929 03:17:50.739995 27663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.739687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:33479 (local address 127.0.0.1:46228)
0929 03:17:50.739851 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.739913 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:33479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 03:17:50.740722 27663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.740501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:33479 (local address 127.0.0.1:46230)
0929 03:17:50.740603 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.740664 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:33479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 03:17:50.740840 27659 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.188:33479: Network error: Client connection negotiation failed: client connection to 127.13.238.188:33479: connect: Connection refused (error 111)
I20260929 03:17:50.753897 27734 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } attempt: 1
I20260929 03:17:50.754858 27743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.754515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:33479 (local address 127.0.0.1:46236)
0929 03:17:50.754710 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.754776 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:33479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 03:17:50.755534 27743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.755342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:33479 (local address 127.0.0.1:46240)
0929 03:17:50.755430 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.755475 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:33479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 03:17:50.755667 27734 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.188:33479: Network error: Client connection negotiation failed: client connection to 127.13.238.188:33479: connect: Connection refused (error 111)
W20260929 03:17:50.811575 27741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:50.811832 27741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:50.811877 27741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:50.815961 27741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:50.816028 27741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:50.816047 27741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:50.816062 27741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:50.816076 27741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:17:50.816015 27659 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } attempt: 2
I20260929 03:17:50.816874 27663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.816524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:33479 (local address 127.0.0.1:46246)
0929 03:17:50.816719 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.816787 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:33479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 03:17:50.817737 27663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.817507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:33479 (local address 127.0.0.1:46252)
0929 03:17:50.817618 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.817674 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:33479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 03:17:50.817857 27659 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.188:33479: Network error: Client connection negotiation failed: client connection to 127.13.238.188:33479: connect: Connection refused (error 111)
I20260929 03:17:50.821673 27741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:35257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:37329,127.13.238.189:46779,127.13.238.188:33479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.188.keytab
--principal=kudu/127.13.238.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.188:33479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.238.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:17:50.822768 27741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:50.823725 27741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:17:50.823808 27734 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } attempt: 2
I20260929 03:17:50.824509 27743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.824260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:33479 (local address 127.0.0.1:46264)
0929 03:17:50.824386 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.824439 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:33479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 03:17:50.825114 27743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.824936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:33479 (local address 127.0.0.1:46268)
0929 03:17:50.825019 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.825059 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:33479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 03:17:50.825227 27734 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.188:33479: Network error: Client connection negotiation failed: client connection to 127.13.238.188:33479: connect: Connection refused (error 111)
W20260929 03:17:50.832226 27751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:50.832239 27748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:50.832228 27749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:50.832415 27741 server_base.cc:1034] running on GCE node
I20260929 03:17:50.833178 27741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:50.834408 27741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:50.835569 27741 hybrid_clock.cc:648] HybridClock initialized: now 1790651870835539 us; error 46 us; skew 500 ppm
Sep 29 03:17:50 dist-test-slave-nbjg krb5kdc[27578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651870, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:50.838332 27741 init.cc:377] Logged in from keytab as kudu/127.13.238.188@KRBTEST.COM (short username kudu)
I20260929 03:17:50.839211 27741 webserver.cc:460] Webserver started at http://127.13.238.188:34481/ using document root <none> and password file <none>
I20260929 03:17:50.839763 27741 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:50.839823 27741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:50.840015 27741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:50.842178 27741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-2/data/instance:
uuid: "8f6285f382fd4aefba5284926ad0b937"
format_stamp: "Formatted at 2026-09-29 03:17:50 on dist-test-slave-nbjg"
server_key: "586b61525bc49bfa8c324675cbedbda7"
server_key_iv: "805b9099384c0228128d6b2a2d6eb993"
server_key_version: "encryptionkey@0"
I20260929 03:17:50.842788 27741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-2/wal/instance:
uuid: "8f6285f382fd4aefba5284926ad0b937"
format_stamp: "Formatted at 2026-09-29 03:17:50 on dist-test-slave-nbjg"
server_key: "586b61525bc49bfa8c324675cbedbda7"
server_key_iv: "805b9099384c0228128d6b2a2d6eb993"
server_key_version: "encryptionkey@0"
I20260929 03:17:50.846508 27741 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:17:50.848856 27758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:50.850075 27741 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:17:50.850181 27741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-2/wal
uuid: "8f6285f382fd4aefba5284926ad0b937"
format_stamp: "Formatted at 2026-09-29 03:17:50 on dist-test-slave-nbjg"
server_key: "586b61525bc49bfa8c324675cbedbda7"
server_key_iv: "805b9099384c0228128d6b2a2d6eb993"
server_key_version: "encryptionkey@0"
I20260929 03:17:50.850310 27741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:50.869467 27741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:50.872464 27741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:50.872618 27741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:50.879309 27741 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.188:33479
I20260929 03:17:50.879623 27810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.188:33479 every 8 connection(s)
I20260929 03:17:50.880275 27741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-2/data/info.pb
I20260929 03:17:50.883179 27811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:50.884584 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 27741
I20260929 03:17:50.884718 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790651713532531-14266-0/minicluster-data/master-2/wal/instance
I20260929 03:17:50.885022 14266 external_mini_cluster.cc:1428] Setting key 72414b7871eeb1d0a6186c5fe1c7978d
I20260929 03:17:50.887542 27811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37329 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:17:50 dist-test-slave-nbjg krb5kdc[27578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.188@KRBTEST.COM
Sep 29 03:17:50 dist-test-slave-nbjg krb5kdc[27578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651870, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.188@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:17:50.897913 27662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.889271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52168 (local address 127.13.238.190:37329)
0929 03:17:50.889412 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:50.889416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:50.889701 (+ 285us) server_negotiation.cc:407] Connection header received
0929 03:17:50.889847 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:50.889850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:50.889899 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:50.889982 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:50.890672 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.891159 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.892081 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.892288 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.892379 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.894919 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:50.894923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:50.894925 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:50.894953 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:50.896148 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.896581 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.896585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.896587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.896661 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.896934 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.896937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.896939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.897140 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:50.897224 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:50.897609 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 03:17:50.897750 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:17:50.898741 27814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.886446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46272 (local address 127.13.238.188:33479)
0929 03:17:50.886887 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:50.886902 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:50.886938 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:17:50.887576 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:50.887594 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:50.887948 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:50.888271 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:50.889122 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.890396 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.891268 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.891736 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.891849 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.894456 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:50.894466 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:50.894481 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:50.894514 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:50.896148 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.896540 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.896546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.896547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.896651 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.896896 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.896899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.896900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.897279 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:50.897461 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:50.897678 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 03:17:50.898336 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":476,"server-negotiator.queue_time_us":282,"thread_start_us":120,"threads_started":1}
I20260929 03:17:50.898743 27815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.889190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:37329 (local address 127.0.0.1:52168)
0929 03:17:50.889568 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.889617 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:50.889738 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:50.890036 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:50.890040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:50.890402 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:50.890548 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:50.890556 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.891276 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.891278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:50.891962 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:50.891972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.892437 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.892441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:50.892532 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.893050 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:50.893068 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:50.894788 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:50.896247 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:50.896256 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:50.896267 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:50.896481 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:50.896740 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:50.896744 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:50.896746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:50.896855 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:50.897237 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:50.897244 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:50.897372 (+ 128us) client_negotiation.cc:769] Sending connection context
0929 03:17:50.897518 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 03:17:50.897846 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":143,"threads_started":1}
I20260929 03:17:50.899988 27811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46779 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:17:50 dist-test-slave-nbjg krb5kdc[27578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651870, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.188@KRBTEST.COM for kudu/127.13.238.189@KRBTEST.COM
Sep 29 03:17:50 dist-test-slave-nbjg krb5kdc[27578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:17:50.907702 27815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.900472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:46779 (local address 127.0.0.1:43730)
0929 03:17:50.900594 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.900608 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:50.900677 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:50.901196 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:50.901199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:50.901337 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:50.901433 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:50.901438 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.902215 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.902219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:50.902720 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:50.902728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.903175 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.903178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:50.903265 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.903681 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:50.903695 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:50.904909 (+ 1214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:50.906385 (+ 1476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:50.906388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:50.906391 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:50.906571 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:50.906812 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:50.906815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:50.906817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:50.906890 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:50.907272 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:50.907274 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:50.907349 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 03:17:50.907451 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 03:17:50.907576 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 03:17:50.907789 27740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.900601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43730 (local address 127.13.238.189:46779)
0929 03:17:50.900721 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:50.900725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:50.900738 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:17:50.900780 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:50.900783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:50.900821 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:50.900882 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:50.901556 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.902090 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.902830 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.903079 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.903165 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.905036 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:50.905040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:50.905042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:50.905067 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:50.906290 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.906658 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.906662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.906663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.906734 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.906989 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.906993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.906995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.907176 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:50.907274 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:50.907501 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 03:17:50.907621 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:17:50.908474 27811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:17:50.908586 27662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.901180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52170 (local address 127.13.238.190:37329)
0929 03:17:50.901299 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:50.901304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:50.901315 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:17:50.901345 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:50.901347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:50.901377 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:50.901442 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:50.902005 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.902519 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.903230 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.903448 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.903537 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.905552 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:50.905555 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:50.905557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:50.905578 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:50.906854 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.907218 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.907222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.907223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.907309 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.907549 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.907552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.907553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.907756 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:50.907986 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:50.908322 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 03:17:50.908433 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 03:17:50 dist-test-slave-nbjg krb5kdc[27578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651870, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.188@KRBTEST.COM for kudu/127.13.238.188@KRBTEST.COM
Sep 29 03:17:50 dist-test-slave-nbjg krb5kdc[27578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.189@KRBTEST.COM
I20260929 03:17:50.916370 27734 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } attempt: 3
I20260929 03:17:50.916750 27815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.908925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:33479 (local address 127.0.0.1:46276)
0929 03:17:50.909015 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.909029 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:50.909094 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:50.909298 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:50.909301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:50.909443 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:50.909501 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:50.909506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.910866 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.910869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:50.911623 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:50.911633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.912177 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.912180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:50.912233 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.912599 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:50.912613 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:50.913773 (+ 1160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:50.915308 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:50.915311 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:50.915313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:50.915533 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:50.915785 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:50.915788 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:50.915789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:50.915881 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:50.916243 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:50.916246 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:50.916360 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 03:17:50.916458 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 03:17:50.916585 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 03:17:50.917007 27659 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } attempt: 3
I20260929 03:17:50.917135 27814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.908972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46276 (local address 127.13.238.188:33479)
0929 03:17:50.909066 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:50.909070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:50.909082 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:17:50.909163 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:50.909166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:50.909204 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:50.909273 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:50.909608 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.910773 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.911725 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.912072 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.912177 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.913897 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:50.913901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:50.913904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:50.913929 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:50.915212 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.915614 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.915617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.915619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.915701 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.915964 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.915967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.915968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.916151 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:50.916224 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:50.916503 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 03:17:50.917009 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":385,"server-negotiator.queue_time_us":39}
I20260929 03:17:50.919613 27740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.909704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43740 (local address 127.13.238.189:46779)
0929 03:17:50.909804 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:50.909807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:50.909841 (+ 34us) server_negotiation.cc:407] Connection header received
0929 03:17:50.909967 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:50.909969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:50.910000 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:50.910055 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:50.910718 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.911466 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.912397 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.912651 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.912755 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.914912 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:50.914915 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:50.914917 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:50.914937 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:50.916323 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.916749 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.916752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.916754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.917959 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.918267 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.918270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.918272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.918470 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:50.918691 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:50.919350 (+ 659us) server_negotiation.cc:299] Negotiation successful
0929 03:17:50.919455 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:17:50.920742 27811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937: Bootstrap starting.
Sep 29 03:17:50 dist-test-slave-nbjg krb5kdc[27578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651870, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.189@KRBTEST.COM for kudu/127.13.238.188@KRBTEST.COM
Sep 29 03:17:50 dist-test-slave-nbjg krb5kdc[27578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651870, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for kudu/127.13.238.188@KRBTEST.COM
I20260929 03:17:50.923377 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:17:50.924173 27811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:50.925109 27811 log.cc:824] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:50.926023 27814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.917020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46284 (local address 127.13.238.188:33479)
0929 03:17:50.917439 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:50.917443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:50.917455 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:17:50.917486 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:50.917489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:50.917528 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:50.918014 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:50.918088 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.918734 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.919796 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.920343 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.920751 (+ 408us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.922631 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:50.922634 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:50.922637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:50.922661 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:50.924379 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.924853 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.924855 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.924857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.924936 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.925245 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.925247 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.925249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.925438 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:50.925563 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:50.925837 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 03:17:50.925918 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":222,"threads_started":1}
I20260929 03:17:50.926077 27743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.916748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:33479 (local address 127.0.0.1:46284)
0929 03:17:50.916870 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.916883 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:50.916956 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:50.917646 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:50.917649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:50.917807 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:50.917917 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:50.917922 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.918853 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.918857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:50.919660 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:50.919671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.920461 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.920464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:50.920555 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.920990 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:50.921002 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:50.922482 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:50.924501 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:50.924504 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:50.924506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:50.924734 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:50.925037 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:50.925040 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:50.925042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:50.925123 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:50.925571 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:50.925574 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:50.925679 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 03:17:50.925797 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 03:17:50.925929 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 03:17:50.929219 27734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c2c6f019b9a4ae793595e8bab202eeb: Bootstrap starting.
I20260929 03:17:50.929433 27811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937: No bootstrap required, opened a new log
I20260929 03:17:50.932112 27734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c2c6f019b9a4ae793595e8bab202eeb: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:50.933212 27734 log.cc:824] T 00000000000000000000000000000000 P 3c2c6f019b9a4ae793595e8bab202eeb: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:50.933652 27811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b4f654fd5c4b999bfca6a69c875a14" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37329 } } peers { permanent_uuid: "3c2c6f019b9a4ae793595e8bab202eeb" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46779 } } peers { permanent_uuid: "8f6285f382fd4aefba5284926ad0b937" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } }
I20260929 03:17:50.933903 27811 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:50.933954 27811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f6285f382fd4aefba5284926ad0b937, State: Initialized, Role: FOLLOWER
I20260929 03:17:50.934511 27811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b4f654fd5c4b999bfca6a69c875a14" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37329 } } peers { permanent_uuid: "3c2c6f019b9a4ae793595e8bab202eeb" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46779 } } peers { permanent_uuid: "8f6285f382fd4aefba5284926ad0b937" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } }
I20260929 03:17:50.935380 27663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.917373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:33479 (local address 127.0.0.1:46294)
0929 03:17:50.917495 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:50.917508 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:50.917580 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:50.918605 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:50.918608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:50.918775 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:50.918886 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:50.918893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.919935 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.919939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:50.920593 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:50.920604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.921363 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.921366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:50.921450 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.921842 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:50.921857 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:50.923376 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:50.926346 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:50.926350 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:50.926352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:50.926604 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:50.933874 (+ 7270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:50.933878 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:50.933881 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:50.933996 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:50.934745 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:50.934748 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:50.934860 (+ 112us) client_negotiation.cc:769] Sending connection context
0929 03:17:50.934951 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 03:17:50.935217 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 03:17:50.935562 27740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.925897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43746 (local address 127.13.238.189:46779)
0929 03:17:50.926000 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:50.926004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:50.927097 (+ 1093us) server_negotiation.cc:407] Connection header received
0929 03:17:50.927268 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:50.927271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:50.927314 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:50.927362 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:50.928279 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.928943 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.930027 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.930350 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.930423 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.932095 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:50.932099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:50.932102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:50.932126 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:50.933728 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.934266 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.934269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.934271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.934351 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.934654 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.934658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.934660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.934866 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:50.934970 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:50.935321 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 03:17:50.935431 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:17:50.935959 27811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:50.936401 27816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.917605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46294 (local address 127.13.238.188:33479)
0929 03:17:50.918381 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:50.918389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:50.918400 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:17:50.918428 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:50.918430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:50.918465 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:50.918986 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:50.919107 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.919809 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.920928 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.921262 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.921347 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.923728 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:50.923732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:50.923735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:50.923758 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:50.925439 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.926696 (+ 1257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.926700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.926701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.926790 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.934274 (+ 7484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.934277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.934279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.934481 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:50.935982 (+ 1501us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:50.936117 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 03:17:50.936229 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"spinlock_wait_cycles":2304}
I20260929 03:17:50.936600 27814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.927165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46308 (local address 127.13.238.188:33479)
0929 03:17:50.927256 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:50.927259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:50.927680 (+ 421us) server_negotiation.cc:407] Connection header received
0929 03:17:50.927827 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:50.927830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:50.927870 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:50.927915 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:50.928758 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.929397 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.930522 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.931066 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.931144 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.933179 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:50.933182 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:50.933185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:50.933210 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:50.934806 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.935369 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.935373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.935374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.935457 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.935787 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.935790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.935792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.935969 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:50.936053 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:50.936374 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 03:17:50.936469 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:17:50.938974 27734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c2c6f019b9a4ae793595e8bab202eeb: No bootstrap required, opened a new log
I20260929 03:17:50.939903 27659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86b4f654fd5c4b999bfca6a69c875a14: Bootstrap starting.
I20260929 03:17:50.940470 27826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b4f654fd5c4b999bfca6a69c875a14" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37329 } } peers { permanent_uuid: "3c2c6f019b9a4ae793595e8bab202eeb" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46779 } } peers { permanent_uuid: "8f6285f382fd4aefba5284926ad0b937" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } } }
I20260929 03:17:50.940577 27826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:17:50.941533 27826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:17:50.941649 27826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b4f654fd5c4b999bfca6a69c875a14" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37329 } } peers { permanent_uuid: "3c2c6f019b9a4ae793595e8bab202eeb" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46779 } } peers { permanent_uuid: "8f6285f382fd4aefba5284926ad0b937" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } }
I20260929 03:17:50.942602 27659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86b4f654fd5c4b999bfca6a69c875a14: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:50.939442 27741 master_runner.cc:186] Error getting master registration for 127.13.238.190:37329: OK, instance_id { permanent_uuid: "86b4f654fd5c4b999bfca6a69c875a14" instance_seqno: 1790651870453991 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:17:50.943080 27741 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:17:50.943437 27659 log.cc:824] T 00000000000000000000000000000000 P 86b4f654fd5c4b999bfca6a69c875a14: Log is configured to *not* fsync() on all Append() calls
W20260929 03:17:50.943514 27837 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:17:50.943571 27837 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:17:50.943523 27734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c2c6f019b9a4ae793595e8bab202eeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b4f654fd5c4b999bfca6a69c875a14" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37329 } } peers { permanent_uuid: "3c2c6f019b9a4ae793595e8bab202eeb" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46779 } } peers { permanent_uuid: "8f6285f382fd4aefba5284926ad0b937" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } }
I20260929 03:17:50.943704 27826 leader_election.cc:288] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86b4f654fd5c4b999bfca6a69c875a14 (127.13.238.190:37329), 3c2c6f019b9a4ae793595e8bab202eeb (127.13.238.189:46779)
I20260929 03:17:50.943722 27734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c2c6f019b9a4ae793595e8bab202eeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:50.943771 27734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c2c6f019b9a4ae793595e8bab202eeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c2c6f019b9a4ae793595e8bab202eeb, State: Initialized, Role: FOLLOWER
I20260929 03:17:50.944260 27734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c2c6f019b9a4ae793595e8bab202eeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b4f654fd5c4b999bfca6a69c875a14" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37329 } } peers { permanent_uuid: "3c2c6f019b9a4ae793595e8bab202eeb" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46779 } } peers { permanent_uuid: "8f6285f382fd4aefba5284926ad0b937" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } }
I20260929 03:17:50.944799 27840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c2c6f019b9a4ae793595e8bab202eeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b4f654fd5c4b999bfca6a69c875a14" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37329 } } peers { permanent_uuid: "3c2c6f019b9a4ae793595e8bab202eeb" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46779 } } peers { permanent_uuid: "8f6285f382fd4aefba5284926ad0b937" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } } }
I20260929 03:17:50.944926 27840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c2c6f019b9a4ae793595e8bab202eeb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:17:50.945715 27734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c2c6f019b9a4ae793595e8bab202eeb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:50.945799 27708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f6285f382fd4aefba5284926ad0b937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c2c6f019b9a4ae793595e8bab202eeb" is_pre_election: true
I20260929 03:17:50.946053 27708 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c2c6f019b9a4ae793595e8bab202eeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f6285f382fd4aefba5284926ad0b937 in term 0.
I20260929 03:17:50.946568 27760 leader_election.cc:302] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c2c6f019b9a4ae793595e8bab202eeb, 8f6285f382fd4aefba5284926ad0b937; no voters:
I20260929 03:17:50.946872 27826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:17:50.947041 27826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:17:50.947155 27826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:50.948460 27826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b4f654fd5c4b999bfca6a69c875a14" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37329 } } peers { permanent_uuid: "3c2c6f019b9a4ae793595e8bab202eeb" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46779 } } peers { permanent_uuid: "8f6285f382fd4aefba5284926ad0b937" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } }
I20260929 03:17:50.949288 27826 leader_election.cc:288] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [CANDIDATE]: Term 1 election: Requested vote from peers 86b4f654fd5c4b999bfca6a69c875a14 (127.13.238.190:37329), 3c2c6f019b9a4ae793595e8bab202eeb (127.13.238.189:46779)
I20260929 03:17:50.949334 27708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f6285f382fd4aefba5284926ad0b937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c2c6f019b9a4ae793595e8bab202eeb"
I20260929 03:17:50.949434 27708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c2c6f019b9a4ae793595e8bab202eeb [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:50.950613 27708 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c2c6f019b9a4ae793595e8bab202eeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f6285f382fd4aefba5284926ad0b937 in term 1.
I20260929 03:17:50.950878 27760 leader_election.cc:302] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c2c6f019b9a4ae793595e8bab202eeb, 8f6285f382fd4aefba5284926ad0b937; no voters:
I20260929 03:17:50.951105 27659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86b4f654fd5c4b999bfca6a69c875a14: No bootstrap required, opened a new log
I20260929 03:17:50.951350 27826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:50.952461 27826 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [term 1 LEADER]: Becoming Leader. State: Replica: 8f6285f382fd4aefba5284926ad0b937, State: Running, Role: LEADER
I20260929 03:17:50.949663 27666 master_runner.cc:186] Error getting master registration for 127.13.238.190:37329: OK, instance_id { permanent_uuid: "86b4f654fd5c4b999bfca6a69c875a14" instance_seqno: 1790651870453991 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:17:50.952913 27666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:17:50.952822 27826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b4f654fd5c4b999bfca6a69c875a14" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37329 } } peers { permanent_uuid: "3c2c6f019b9a4ae793595e8bab202eeb" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46779 } } peers { permanent_uuid: "8f6285f382fd4aefba5284926ad0b937" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } }
I20260929 03:17:50.955215 27659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86b4f654fd5c4b999bfca6a69c875a14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b4f654fd5c4b999bfca6a69c875a14" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37329 } } peers { permanent_uuid: "3c2c6f019b9a4ae793595e8bab202eeb" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46779 } } peers { permanent_uuid: "8f6285f382fd4aefba5284926ad0b937" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } }
I20260929 03:17:50.955463 27659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86b4f654fd5c4b999bfca6a69c875a14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:50.955523 27659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86b4f654fd5c4b999bfca6a69c875a14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b4f654fd5c4b999bfca6a69c875a14, State: Initialized, Role: FOLLOWER
I20260929 03:17:50.955276 27852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f6285f382fd4aefba5284926ad0b937. Latest consensus state: current_term: 1 leader_uuid: "8f6285f382fd4aefba5284926ad0b937" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b4f654fd5c4b999bfca6a69c875a14" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37329 } } peers { permanent_uuid: "3c2c6f019b9a4ae793595e8bab202eeb" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46779 } } peers { permanent_uuid: "8f6285f382fd4aefba5284926ad0b937" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } } }
I20260929 03:17:50.955658 27852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:50.956029 27659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86b4f654fd5c4b999bfca6a69c875a14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b4f654fd5c4b999bfca6a69c875a14" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37329 } } peers { permanent_uuid: "3c2c6f019b9a4ae793595e8bab202eeb" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46779 } } peers { permanent_uuid: "8f6285f382fd4aefba5284926ad0b937" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } }
W20260929 03:17:50.956528 27851 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c2c6f019b9a4ae793595e8bab202eeb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:17:50.956597 27851 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:17:50.956683 27856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86b4f654fd5c4b999bfca6a69c875a14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b4f654fd5c4b999bfca6a69c875a14" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37329 } } peers { permanent_uuid: "3c2c6f019b9a4ae793595e8bab202eeb" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46779 } } peers { permanent_uuid: "8f6285f382fd4aefba5284926ad0b937" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } } }
I20260929 03:17:50.956944 27856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86b4f654fd5c4b999bfca6a69c875a14 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:17:50.957399 27659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86b4f654fd5c4b999bfca6a69c875a14 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:50.957444 27633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f6285f382fd4aefba5284926ad0b937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86b4f654fd5c4b999bfca6a69c875a14" is_pre_election: true
I20260929 03:17:50.957720 27633 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 86b4f654fd5c4b999bfca6a69c875a14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f6285f382fd4aefba5284926ad0b937 in term 0.
I20260929 03:17:50.957988 27632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f6285f382fd4aefba5284926ad0b937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86b4f654fd5c4b999bfca6a69c875a14"
I20260929 03:17:50.958104 27632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86b4f654fd5c4b999bfca6a69c875a14 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:50.958292 27855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:17:50.959295 27855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:17:50.959444 27632 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 86b4f654fd5c4b999bfca6a69c875a14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f6285f382fd4aefba5284926ad0b937 in term 1.
I20260929 03:17:50.960417 27662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:50.926338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52176 (local address 127.13.238.190:37329)
0929 03:17:50.926593 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:50.926597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:50.943938 (+ 17341us) server_negotiation.cc:407] Connection header received
0929 03:17:50.944101 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:50.944104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:50.944152 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:50.944199 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:50.945280 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.946027 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.947177 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:50.947482 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:50.947582 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:50.950319 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:50.950323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:50.950326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:50.950353 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:50.952075 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.958838 (+ 6763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.958843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.958844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.958952 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:50.959335 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:50.959339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:50.959341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:50.959572 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:50.959690 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:50.960124 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 03:17:50.960257 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:17:50.965106 27708 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c2c6f019b9a4ae793595e8bab202eeb [term 1 FOLLOWER]: Refusing update from remote peer 8f6285f382fd4aefba5284926ad0b937: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:50.965209 27632 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 86b4f654fd5c4b999bfca6a69c875a14 [term 1 FOLLOWER]: Refusing update from remote peer 8f6285f382fd4aefba5284926ad0b937: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:50.965807 27852 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c2c6f019b9a4ae793595e8bab202eeb" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:50.967568 27852 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86b4f654fd5c4b999bfca6a69c875a14" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:50.970423 27840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c2c6f019b9a4ae793595e8bab202eeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f6285f382fd4aefba5284926ad0b937. Latest consensus state: current_term: 1 leader_uuid: "8f6285f382fd4aefba5284926ad0b937" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b4f654fd5c4b999bfca6a69c875a14" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37329 } } peers { permanent_uuid: "3c2c6f019b9a4ae793595e8bab202eeb" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46779 } } peers { permanent_uuid: "8f6285f382fd4aefba5284926ad0b937" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } } }
I20260929 03:17:50.970548 27840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c2c6f019b9a4ae793595e8bab202eeb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:17:50.970996 27856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86b4f654fd5c4b999bfca6a69c875a14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f6285f382fd4aefba5284926ad0b937. Latest consensus state: current_term: 1 leader_uuid: "8f6285f382fd4aefba5284926ad0b937" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b4f654fd5c4b999bfca6a69c875a14" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37329 } } peers { permanent_uuid: "3c2c6f019b9a4ae793595e8bab202eeb" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46779 } } peers { permanent_uuid: "8f6285f382fd4aefba5284926ad0b937" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } } }
I20260929 03:17:50.971117 27856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86b4f654fd5c4b999bfca6a69c875a14 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:17:50.973259 27826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f6285f382fd4aefba5284926ad0b937" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b4f654fd5c4b999bfca6a69c875a14" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37329 } } peers { permanent_uuid: "3c2c6f019b9a4ae793595e8bab202eeb" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46779 } } peers { permanent_uuid: "8f6285f382fd4aefba5284926ad0b937" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } } }
I20260929 03:17:50.973372 27826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:50.973526 27859 mvcc.cc:205] Tried to move back new op lower bound from 7334510063466266624 to 7334510063426203648. Current Snapshot: MvccSnapshot[applied={T|T < 7334510063466266624}]
I20260929 03:17:50.973614 27856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86b4f654fd5c4b999bfca6a69c875a14 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f6285f382fd4aefba5284926ad0b937" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b4f654fd5c4b999bfca6a69c875a14" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37329 } } peers { permanent_uuid: "3c2c6f019b9a4ae793595e8bab202eeb" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46779 } } peers { permanent_uuid: "8f6285f382fd4aefba5284926ad0b937" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } } }
I20260929 03:17:50.973726 27856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86b4f654fd5c4b999bfca6a69c875a14 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:17:50.974812 27855 catalog_manager.cc:1347] Generated new cluster ID: 27725e7b6d6849b88d4e436861456205
I20260929 03:17:50.974879 27855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
W20260929 03:17:50.978502 27868 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 86b4f654fd5c4b999bfca6a69c875a14: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:17:50.978574 27868 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:17:50.981868 27852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f6285f382fd4aefba5284926ad0b937" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b4f654fd5c4b999bfca6a69c875a14" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37329 } } peers { permanent_uuid: "3c2c6f019b9a4ae793595e8bab202eeb" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46779 } } peers { permanent_uuid: "8f6285f382fd4aefba5284926ad0b937" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } } }
I20260929 03:17:50.981978 27852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:50.983064 27840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c2c6f019b9a4ae793595e8bab202eeb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f6285f382fd4aefba5284926ad0b937" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b4f654fd5c4b999bfca6a69c875a14" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 37329 } } peers { permanent_uuid: "3c2c6f019b9a4ae793595e8bab202eeb" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 46779 } } peers { permanent_uuid: "8f6285f382fd4aefba5284926ad0b937" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 33479 } } }
I20260929 03:17:50.983168 27840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c2c6f019b9a4ae793595e8bab202eeb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:17:50.983335 27869 mvcc.cc:205] Tried to move back new op lower bound from 7334510063466266624 to 7334510063426203648. Current Snapshot: MvccSnapshot[applied={T|T < 7334510063466266624}]
I20260929 03:17:50.987493 27870 mvcc.cc:205] Tried to move back new op lower bound from 7334510063466266624 to 7334510063426203648. Current Snapshot: MvccSnapshot[applied={T|T < 7334510063466266624}]
I20260929 03:17:51.010942 27855 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:17:51.012301 27855 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:17:51.023375 27855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f6285f382fd4aefba5284926ad0b937: Generated new TSK 0
I20260929 03:17:51.024369 27855 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:17:51.957973 27851 catalog_manager.cc:1259] Loaded cluster ID: 27725e7b6d6849b88d4e436861456205
I20260929 03:17:51.958060 27851 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3c2c6f019b9a4ae793595e8bab202eeb: loading cluster ID for follower catalog manager: success
I20260929 03:17:51.959798 27851 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3c2c6f019b9a4ae793595e8bab202eeb: acquiring CA information for follower catalog manager: success
I20260929 03:17:51.960925 27851 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3c2c6f019b9a4ae793595e8bab202eeb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:17:51.979823 27868 catalog_manager.cc:1259] Loaded cluster ID: 27725e7b6d6849b88d4e436861456205
I20260929 03:17:51.979897 27868 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 86b4f654fd5c4b999bfca6a69c875a14: loading cluster ID for follower catalog manager: success
I20260929 03:17:51.981503 27868 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 86b4f654fd5c4b999bfca6a69c875a14: acquiring CA information for follower catalog manager: success
I20260929 03:17:51.982682 27868 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 86b4f654fd5c4b999bfca6a69c875a14: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:17:52.082743 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 27591
I20260929 03:17:52.088618 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 27666
I20260929 03:17:52.093837 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 27741
2026-09-29T03:17:52Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2912 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:17:52 dist-test-slave-nbjg krb5kdc[27884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:17:52 dist-test-slave-nbjg krb5kdc[27884](info): set up 2 sockets
Sep 29 03:17:52 dist-test-slave-nbjg krb5kdc[27884](info): commencing operation
krb5kdc: starting...
W20260929 03:17:53.070452 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:17:53 dist-test-slave-nbjg krb5kdc[27884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:17:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:17:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:17:53.189656 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:34583
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:40329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:34583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:53.301978 27897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:53.302297 27897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:53.302348 27897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:53.306331 27897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:53.306391 27897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:53.306408 27897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:53.306424 27897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:53.306438 27897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:17:53.311786 27897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:40329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:34583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:34583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:17:53.312848 27897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:53.313898 27897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:53.322233 27902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:53.322404 27897 server_base.cc:1034] running on GCE node
W20260929 03:17:53.322237 27905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:53.322548 27903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:53.323099 27897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:53.324188 27897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:53.325346 27897 hybrid_clock.cc:648] HybridClock initialized: now 1790651873325328 us; error 29 us; skew 500 ppm
Sep 29 03:17:53 dist-test-slave-nbjg krb5kdc[27884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651873, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:53.327939 27897 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:17:53.328820 27897 webserver.cc:460] Webserver started at http://127.13.238.190:43429/ using document root <none> and password file <none>
I20260929 03:17:53.329350 27897 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:53.329411 27897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:53.329617 27897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:53.331713 27897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "ded7ec78eba94f77a85767488528ed13"
format_stamp: "Formatted at 2026-09-29 03:17:53 on dist-test-slave-nbjg"
server_key: "e7c71efd5eff9a27022ae0062da1ac08"
server_key_iv: "e803a8521f287e27e2d4700ab4fa03b2"
server_key_version: "encryptionkey@0"
I20260929 03:17:53.332248 27897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "ded7ec78eba94f77a85767488528ed13"
format_stamp: "Formatted at 2026-09-29 03:17:53 on dist-test-slave-nbjg"
server_key: "e7c71efd5eff9a27022ae0062da1ac08"
server_key_iv: "e803a8521f287e27e2d4700ab4fa03b2"
server_key_version: "encryptionkey@0"
I20260929 03:17:53.335911 27897 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260929 03:17:53.338305 27912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:53.339474 27897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:17:53.339579 27897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "ded7ec78eba94f77a85767488528ed13"
format_stamp: "Formatted at 2026-09-29 03:17:53 on dist-test-slave-nbjg"
server_key: "e7c71efd5eff9a27022ae0062da1ac08"
server_key_iv: "e803a8521f287e27e2d4700ab4fa03b2"
server_key_version: "encryptionkey@0"
I20260929 03:17:53.339679 27897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:53.353650 27897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:53.356503 27897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:53.356671 27897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:53.363512 27897 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:34583
I20260929 03:17:53.363541 27964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:34583 every 8 connection(s)
I20260929 03:17:53.364449 27897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:17:53.365612 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 27897
I20260929 03:17:53.365734 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:17:53.366052 14266 external_mini_cluster.cc:1428] Setting key cded34d774d5b00d2800ca2c078b8622
I20260929 03:17:53.367660 27965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:53.373896 27965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ded7ec78eba94f77a85767488528ed13: Bootstrap starting.
Sep 29 03:17:53 dist-test-slave-nbjg krb5kdc[27884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:17:53.376348 27965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ded7ec78eba94f77a85767488528ed13: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:53.377099 27965 log.cc:824] T 00000000000000000000000000000000 P ded7ec78eba94f77a85767488528ed13: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:53.379173 27965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ded7ec78eba94f77a85767488528ed13: No bootstrap required, opened a new log
I20260929 03:17:53.379575 27968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:53.367346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40654 (local address 127.13.238.190:34583)
0929 03:17:53.367759 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:53.367777 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:53.367818 (+ 41us) server_negotiation.cc:407] Connection header received
0929 03:17:53.368444 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:53.368461 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:53.368794 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:53.369068 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:53.369924 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:53.370940 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:53.372035 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:53.372488 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:53.372599 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:53.375257 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:53.375263 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:53.375276 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:53.375303 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:53.376833 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:53.377318 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:53.377323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:53.377327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:53.377438 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:53.377697 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:53.377700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:53.377702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:53.378113 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:53.378296 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:53.378557 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 03:17:53.378848 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":113,"threads_started":1}
I20260929 03:17:53.382247 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:17:53.382783 27965 raft_consensus.cc:348] T 00000000000000000000000000000000 P ded7ec78eba94f77a85767488528ed13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded7ec78eba94f77a85767488528ed13" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 34583 } }
I20260929 03:17:53.383055 27965 raft_consensus.cc:374] T 00000000000000000000000000000000 P ded7ec78eba94f77a85767488528ed13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:53.383118 27965 raft_consensus.cc:729] T 00000000000000000000000000000000 P ded7ec78eba94f77a85767488528ed13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded7ec78eba94f77a85767488528ed13, State: Initialized, Role: FOLLOWER
I20260929 03:17:53.383543 27965 consensus_queue.cc:260] T 00000000000000000000000000000000 P ded7ec78eba94f77a85767488528ed13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded7ec78eba94f77a85767488528ed13" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 34583 } }
I20260929 03:17:53.383673 27965 raft_consensus.cc:388] T 00000000000000000000000000000000 P ded7ec78eba94f77a85767488528ed13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:17:53.383736 27965 raft_consensus.cc:482] T 00000000000000000000000000000000 P ded7ec78eba94f77a85767488528ed13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:17:53.383834 27965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ded7ec78eba94f77a85767488528ed13 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:53.384699 27965 raft_consensus.cc:504] T 00000000000000000000000000000000 P ded7ec78eba94f77a85767488528ed13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded7ec78eba94f77a85767488528ed13" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 34583 } }
I20260929 03:17:53.385002 27965 leader_election.cc:302] T 00000000000000000000000000000000 P ded7ec78eba94f77a85767488528ed13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ded7ec78eba94f77a85767488528ed13; no voters:
I20260929 03:17:53.385272 27965 leader_election.cc:288] T 00000000000000000000000000000000 P ded7ec78eba94f77a85767488528ed13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:17:53.385375 27970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ded7ec78eba94f77a85767488528ed13 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:53.385581 27970 raft_consensus.cc:686] T 00000000000000000000000000000000 P ded7ec78eba94f77a85767488528ed13 [term 1 LEADER]: Becoming Leader. State: Replica: ded7ec78eba94f77a85767488528ed13, State: Running, Role: LEADER
I20260929 03:17:53.385855 27970 consensus_queue.cc:237] T 00000000000000000000000000000000 P ded7ec78eba94f77a85767488528ed13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded7ec78eba94f77a85767488528ed13" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 34583 } }
I20260929 03:17:53.386447 27965 sys_catalog.cc:564] T 00000000000000000000000000000000 P ded7ec78eba94f77a85767488528ed13 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:53.387542 27971 sys_catalog.cc:455] T 00000000000000000000000000000000 P ded7ec78eba94f77a85767488528ed13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ded7ec78eba94f77a85767488528ed13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded7ec78eba94f77a85767488528ed13" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 34583 } } }
I20260929 03:17:53.387673 27971 sys_catalog.cc:458] T 00000000000000000000000000000000 P ded7ec78eba94f77a85767488528ed13 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:53.387540 27972 sys_catalog.cc:455] T 00000000000000000000000000000000 P ded7ec78eba94f77a85767488528ed13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ded7ec78eba94f77a85767488528ed13. Latest consensus state: current_term: 1 leader_uuid: "ded7ec78eba94f77a85767488528ed13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded7ec78eba94f77a85767488528ed13" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 34583 } } }
I20260929 03:17:53.387961 27972 sys_catalog.cc:458] T 00000000000000000000000000000000 P ded7ec78eba94f77a85767488528ed13 [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:53.388535 27979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:17:53.391095 27979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:17:53.396378 27979 catalog_manager.cc:1347] Generated new cluster ID: 2c2086dadebf4e78935d58075098b851
I20260929 03:17:53.396440 27979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:17:53.401466 27979 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:17:53.402673 27979 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:17:53.409828 27979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ded7ec78eba94f77a85767488528ed13: Generated new TSK 0
I20260929 03:17:53.410678 27979 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:17:53.438084 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 27897
2026-09-29T03:17:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1343 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:17:53 dist-test-slave-nbjg krb5kdc[27992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:17:53 dist-test-slave-nbjg krb5kdc[27992](info): set up 2 sockets
Sep 29 03:17:53 dist-test-slave-nbjg krb5kdc[27992](info): commencing operation
krb5kdc: starting...
W20260929 03:17:54.394505 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:17:54 dist-test-slave-nbjg krb5kdc[27992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:17:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:17:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:17:54.510833 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:43239
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:43239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:54.617437 28005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:54.617704 28005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:54.617756 28005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:54.621704 28005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:54.621759 28005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:54.621785 28005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:54.621805 28005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:54.621825 28005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:17:54.627050 28005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:43239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:43239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:17:54.628072 28005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:54.629066 28005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:54.637475 28011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:54.637504 28010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:54.637509 28013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:54.637694 28005 server_base.cc:1034] running on GCE node
I20260929 03:17:54.638424 28005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:54.639495 28005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:54.640704 28005 hybrid_clock.cc:648] HybridClock initialized: now 1790651874640682 us; error 33 us; skew 500 ppm
Sep 29 03:17:54 dist-test-slave-nbjg krb5kdc[27992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651874, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:54.646405 28005 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:17:54.647326 28005 webserver.cc:460] Webserver started at http://127.13.238.190:43197/ using document root <none> and password file <none>
I20260929 03:17:54.647861 28005 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:54.647924 28005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:54.648132 28005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:54.650156 28005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "678678b2d3d24b57b34c646855d8b52d"
format_stamp: "Formatted at 2026-09-29 03:17:54 on dist-test-slave-nbjg"
server_key: "801aa173609a1d676c827bde99fe34b8"
server_key_iv: "35be0706771725e6669041506764235e"
server_key_version: "encryptionkey@0"
I20260929 03:17:54.650739 28005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "678678b2d3d24b57b34c646855d8b52d"
format_stamp: "Formatted at 2026-09-29 03:17:54 on dist-test-slave-nbjg"
server_key: "801aa173609a1d676c827bde99fe34b8"
server_key_iv: "35be0706771725e6669041506764235e"
server_key_version: "encryptionkey@0"
I20260929 03:17:54.654382 28005 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:17:54.656812 28020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:54.658113 28005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:17:54.658244 28005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "678678b2d3d24b57b34c646855d8b52d"
format_stamp: "Formatted at 2026-09-29 03:17:54 on dist-test-slave-nbjg"
server_key: "801aa173609a1d676c827bde99fe34b8"
server_key_iv: "35be0706771725e6669041506764235e"
server_key_version: "encryptionkey@0"
I20260929 03:17:54.658339 28005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:54.680739 28005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:54.683843 28005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:54.683997 28005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:54.690485 28005 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:43239
I20260929 03:17:54.690826 28072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:43239 every 8 connection(s)
I20260929 03:17:54.691495 28005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:17:54.694274 28073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:54.696636 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 28005
I20260929 03:17:54.696748 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:17:54.697014 14266 external_mini_cluster.cc:1428] Setting key aa308b594ab0374d46a851f4b3d41e92
I20260929 03:17:54.700481 28073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d: Bootstrap starting.
I20260929 03:17:54.703167 28073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:54.703984 28073 log.cc:824] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:54.706005 28073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d: No bootstrap required, opened a new log
Sep 29 03:17:54 dist-test-slave-nbjg krb5kdc[27992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:17:54.710738 28073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678678b2d3d24b57b34c646855d8b52d" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43239 } }
I20260929 03:17:54.711046 28073 raft_consensus.cc:374] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:54.711090 28073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 678678b2d3d24b57b34c646855d8b52d, State: Initialized, Role: FOLLOWER
I20260929 03:17:54.711604 28073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678678b2d3d24b57b34c646855d8b52d" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43239 } }
I20260929 03:17:54.711779 28073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:17:54.711850 28073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:17:54.711949 28073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:54.713066 28076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:54.698437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39124 (local address 127.13.238.190:43239)
0929 03:17:54.698865 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:54.698884 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:54.698928 (+ 44us) server_negotiation.cc:407] Connection header received
0929 03:17:54.699739 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:54.699763 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:54.700189 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:54.700551 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:54.701233 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:54.702733 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:54.703656 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:54.704349 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:54.704491 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:54.707232 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:54.707238 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:54.707256 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:54.707289 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:54.709298 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:54.709745 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:54.709751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:54.709757 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:54.709902 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:54.710252 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:54.710255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:54.710257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:54.710792 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:54.710986 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:54.711621 (+ 635us) server_negotiation.cc:299] Negotiation successful
0929 03:17:54.712051 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":110,"threads_started":1}
I20260929 03:17:54.713092 28073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678678b2d3d24b57b34c646855d8b52d" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43239 } }
I20260929 03:17:54.713442 28073 leader_election.cc:302] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 678678b2d3d24b57b34c646855d8b52d; no voters:
I20260929 03:17:54.713790 28073 leader_election.cc:288] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:17:54.713882 28078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:54.714124 28078 raft_consensus.cc:686] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [term 1 LEADER]: Becoming Leader. State: Replica: 678678b2d3d24b57b34c646855d8b52d, State: Running, Role: LEADER
I20260929 03:17:54.714459 28078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678678b2d3d24b57b34c646855d8b52d" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43239 } }
I20260929 03:17:54.715083 28073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:54.716032 28079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "678678b2d3d24b57b34c646855d8b52d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678678b2d3d24b57b34c646855d8b52d" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43239 } } }
I20260929 03:17:54.716161 28079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:54.716476 28087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:17:54.716435 28080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 678678b2d3d24b57b34c646855d8b52d. Latest consensus state: current_term: 1 leader_uuid: "678678b2d3d24b57b34c646855d8b52d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678678b2d3d24b57b34c646855d8b52d" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43239 } } }
I20260929 03:17:54.716544 28080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:54.719614 28087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:17:54.725153 28087 catalog_manager.cc:1347] Generated new cluster ID: 6655e4cd99034f35a1cc590422e33d7e
I20260929 03:17:54.725221 28087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:17:54.736348 28087 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:17:54.737572 28087 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:17:54.752043 28087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d: Generated new TSK 0
I20260929 03:17:54.752795 28087 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
I20260929 03:17:54.788594 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:0
--local_ip_for_outbound_sockets=127.13.238.129
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43239
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:54.927243 28099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:54.927495 28099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:54.927536 28099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:54.931492 28099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:54.931551 28099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:54.931612 28099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:17:54.936883 28099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:17:54.937934 28099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:54.938993 28099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:17:54.948284 28099 server_base.cc:1034] running on GCE node
W20260929 03:17:54.948304 28104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:54.948295 28107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:54.948263 28105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:54.948918 28099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:54.949522 28099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:54.950676 28099 hybrid_clock.cc:648] HybridClock initialized: now 1790651874950655 us; error 34 us; skew 500 ppm
Sep 29 03:17:54 dist-test-slave-nbjg krb5kdc[27992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651874, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:54.953276 28099 init.cc:377] Logged in from keytab as kudu/127.13.238.129@KRBTEST.COM (short username kudu)
I20260929 03:17:54.954046 28099 webserver.cc:460] Webserver started at http://127.13.238.129:45851/ using document root <none> and password file <none>
I20260929 03:17:54.954595 28099 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:54.954659 28099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:54.954873 28099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:54.956825 28099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/data/instance:
uuid: "7ed14d80f04c482bb017fd772a1c22ae"
format_stamp: "Formatted at 2026-09-29 03:17:54 on dist-test-slave-nbjg"
server_key: "0203449a79a66624da04a0c128d8c1bf"
server_key_iv: "421357bffe42a88dedb6260c2ab2491f"
server_key_version: "encryptionkey@0"
I20260929 03:17:54.957360 28099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance:
uuid: "7ed14d80f04c482bb017fd772a1c22ae"
format_stamp: "Formatted at 2026-09-29 03:17:54 on dist-test-slave-nbjg"
server_key: "0203449a79a66624da04a0c128d8c1bf"
server_key_iv: "421357bffe42a88dedb6260c2ab2491f"
server_key_version: "encryptionkey@0"
I20260929 03:17:54.960852 28099 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 03:17:54.963290 28114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:54.964532 28099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:17:54.964635 28099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "7ed14d80f04c482bb017fd772a1c22ae"
format_stamp: "Formatted at 2026-09-29 03:17:54 on dist-test-slave-nbjg"
server_key: "0203449a79a66624da04a0c128d8c1bf"
server_key_iv: "421357bffe42a88dedb6260c2ab2491f"
server_key_version: "encryptionkey@0"
I20260929 03:17:54.964722 28099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:54.988858 28099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:54.991930 28099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:54.992113 28099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:54.992699 28099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:54.993538 28099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:54.993582 28099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:54.993643 28099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:54.993670 28099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:55.002149 28099 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:42287
I20260929 03:17:55.002182 28227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:42287 every 8 connection(s)
I20260929 03:17:55.003065 28099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:17:55.005645 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 28099
I20260929 03:17:55.005777 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance
I20260929 03:17:55.006083 14266 external_mini_cluster.cc:1428] Setting key 28296eb0538c4c0ef02e8aeb02f2eb95
Sep 29 03:17:55 dist-test-slave-nbjg krb5kdc[27992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651874, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.238.130@KRBTEST.COM; defaulting to no policy
I20260929 03:17:55.017060 28076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:55.004922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:34377 (local address 127.13.238.190:43239)
0929 03:17:55.005057 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:55.005061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:55.005942 (+ 881us) server_negotiation.cc:407] Connection header received
0929 03:17:55.007420 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:55.007423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:55.007472 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:55.007560 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:55.009036 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.009578 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.010797 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.011045 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.011134 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:55.013680 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:55.013683 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:55.013686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:55.013710 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:55.015172 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:55.015646 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:55.015649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:55.015651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:55.015727 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:55.016025 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:55.016027 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:55.016028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:55.016247 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:55.016327 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:55.016802 (+ 475us) server_negotiation.cc:299] Negotiation successful
0929 03:17:55.016910 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.130@KRBTEST.COM" created.
I20260929 03:17:55.018038 28230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:55.005290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.129:34377)
0929 03:17:55.005774 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:55.005828 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:55.007207 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:55.007757 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:55.007767 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:55.008234 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:55.008908 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:55.008926 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.009725 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.009728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:55.010680 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:55.010691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.011198 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.011202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:55.011357 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:55.011921 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:55.011937 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:55.013503 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:55.015289 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:55.015294 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:55.015307 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:55.015564 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:55.015824 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:55.015827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:55.015829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:55.015965 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:55.016343 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:55.016349 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:55.016598 (+ 249us) client_negotiation.cc:769] Sending connection context
0929 03:17:55.016753 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 03:17:55.017159 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":121,"threads_started":1}
I20260929 03:17:55.019331 28228 heartbeater.cc:347] Connected to a master server at 127.13.238.190:43239
I20260929 03:17:55.019582 28228 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:55.020084 28228 heartbeater.cc:511] Master 127.13.238.190:43239 requested a full tablet report, sending...
I20260929 03:17:55.021627 28037 ts_manager.cc:194] Registered new tserver with Master: 7ed14d80f04c482bb017fd772a1c22ae (127.13.238.129:42287)
I20260929 03:17:55.022790 28037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.129@KRBTEST.COM'} at 127.13.238.129:34377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
I20260929 03:17:55.033524 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:0
--local_ip_for_outbound_sockets=127.13.238.130
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43239
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:55.141831 28233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:55.142096 28233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:55.142153 28233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:55.146138 28233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:55.146216 28233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:55.146292 28233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:17:55.151535 28233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:17:55.152563 28233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:55.153604 28233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:55.163322 28238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:55.163364 28239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:55.163388 28241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:55.163355 28233 server_base.cc:1034] running on GCE node
I20260929 03:17:55.163868 28233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:55.164525 28233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:55.165681 28233 hybrid_clock.cc:648] HybridClock initialized: now 1790651875165659 us; error 34 us; skew 500 ppm
Sep 29 03:17:55 dist-test-slave-nbjg krb5kdc[27992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651875, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:55.168660 28233 init.cc:377] Logged in from keytab as kudu/127.13.238.130@KRBTEST.COM (short username kudu)
I20260929 03:17:55.169502 28233 webserver.cc:460] Webserver started at http://127.13.238.130:45469/ using document root <none> and password file <none>
I20260929 03:17:55.170025 28233 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:55.170081 28233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:55.170290 28233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:55.172284 28233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/data/instance:
uuid: "c92aa75a5e8e4eb2b97286955650f3f4"
format_stamp: "Formatted at 2026-09-29 03:17:55 on dist-test-slave-nbjg"
server_key: "6e0af4329db494a53b431ed7f300d266"
server_key_iv: "ed02205b4b588768e83f79a1a452a9c7"
server_key_version: "encryptionkey@0"
I20260929 03:17:55.172807 28233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance:
uuid: "c92aa75a5e8e4eb2b97286955650f3f4"
format_stamp: "Formatted at 2026-09-29 03:17:55 on dist-test-slave-nbjg"
server_key: "6e0af4329db494a53b431ed7f300d266"
server_key_iv: "ed02205b4b588768e83f79a1a452a9c7"
server_key_version: "encryptionkey@0"
I20260929 03:17:55.176649 28233 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:17:55.179019 28248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:55.180202 28233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 03:17:55.180305 28233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "c92aa75a5e8e4eb2b97286955650f3f4"
format_stamp: "Formatted at 2026-09-29 03:17:55 on dist-test-slave-nbjg"
server_key: "6e0af4329db494a53b431ed7f300d266"
server_key_iv: "ed02205b4b588768e83f79a1a452a9c7"
server_key_version: "encryptionkey@0"
I20260929 03:17:55.180388 28233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:55.211750 28233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:55.214587 28233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:55.214740 28233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:55.215377 28233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:55.216275 28233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:55.216320 28233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:55.216359 28233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:55.216379 28233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:55.224481 28233 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:41871
I20260929 03:17:55.224532 28361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:41871 every 8 connection(s)
I20260929 03:17:55.225335 28233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:17:55.229719 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 28233
I20260929 03:17:55.229832 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance
I20260929 03:17:55.230144 14266 external_mini_cluster.cc:1428] Setting key 4420de18b79ebe8f116934fdd92af84c
Sep 29 03:17:55 dist-test-slave-nbjg krb5kdc[27992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651875, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.238.131@KRBTEST.COM; defaulting to no policy
I20260929 03:17:55.239540 28076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:55.227144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:57835 (local address 127.13.238.190:43239)
0929 03:17:55.227280 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:55.227285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:55.228160 (+ 875us) server_negotiation.cc:407] Connection header received
0929 03:17:55.229477 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:55.229480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:55.229529 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:55.229606 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:55.231043 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.231773 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.233319 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.233667 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.233772 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:55.236208 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:55.236211 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:55.236214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:55.236238 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:55.237516 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:55.238005 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:55.238007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:55.238009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:55.238084 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:55.238431 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:55.238435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:55.238437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:55.238620 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:55.238707 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:55.239289 (+ 582us) server_negotiation.cc:299] Negotiation successful
0929 03:17:55.239413 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 03:17:55.240406 28364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:55.227570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.130:57835)
0929 03:17:55.228013 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:55.228056 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:55.229257 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:55.229823 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:55.229833 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:55.230370 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:55.230915 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:55.230929 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.231902 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.231905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:55.233196 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:55.233207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.233815 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.233818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:55.233968 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:55.234543 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:55.234560 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:55.236037 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:55.237626 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:55.237632 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:55.237650 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:55.237909 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:55.238166 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:55.238169 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:55.238171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:55.238343 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:55.238727 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:55.238734 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:55.238987 (+ 253us) client_negotiation.cc:769] Sending connection context
0929 03:17:55.239190 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 03:17:55.239468 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":127,"threads_started":1}
I20260929 03:17:55.241532 28362 heartbeater.cc:347] Connected to a master server at 127.13.238.190:43239
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.131@KRBTEST.COM" created.
I20260929 03:17:55.241822 28362 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:55.242368 28362 heartbeater.cc:511] Master 127.13.238.190:43239 requested a full tablet report, sending...
I20260929 03:17:55.243521 28037 ts_manager.cc:194] Registered new tserver with Master: c92aa75a5e8e4eb2b97286955650f3f4 (127.13.238.130:41871)
I20260929 03:17:55.244095 28037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.130@KRBTEST.COM'} at 127.13.238.130:57835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
I20260929 03:17:55.256301 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:0
--local_ip_for_outbound_sockets=127.13.238.131
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43239
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:55.373150 28367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:55.373404 28367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:55.373445 28367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:55.377447 28367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:55.377502 28367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:55.377559 28367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:17:55.382787 28367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:17:55.383787 28367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:55.384845 28367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:55.394027 28372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:55.394063 28375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:55.394282 28367 server_base.cc:1034] running on GCE node
W20260929 03:17:55.394039 28373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:55.394702 28367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:55.395334 28367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:55.396476 28367 hybrid_clock.cc:648] HybridClock initialized: now 1790651875396451 us; error 38 us; skew 500 ppm
Sep 29 03:17:55 dist-test-slave-nbjg krb5kdc[27992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651875, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:55.399085 28367 init.cc:377] Logged in from keytab as kudu/127.13.238.131@KRBTEST.COM (short username kudu)
I20260929 03:17:55.399940 28367 webserver.cc:460] Webserver started at http://127.13.238.131:39617/ using document root <none> and password file <none>
I20260929 03:17:55.400467 28367 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:55.400525 28367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:55.400717 28367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:17:55.402727 28367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/data/instance:
uuid: "c231762c0bfe4a319fa0c14a1606623c"
format_stamp: "Formatted at 2026-09-29 03:17:55 on dist-test-slave-nbjg"
server_key: "a3c0f0e805a2d00fa717ad17433d09aa"
server_key_iv: "b004142a762f46847541a3249a903237"
server_key_version: "encryptionkey@0"
I20260929 03:17:55.403257 28367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance:
uuid: "c231762c0bfe4a319fa0c14a1606623c"
format_stamp: "Formatted at 2026-09-29 03:17:55 on dist-test-slave-nbjg"
server_key: "a3c0f0e805a2d00fa717ad17433d09aa"
server_key_iv: "b004142a762f46847541a3249a903237"
server_key_version: "encryptionkey@0"
I20260929 03:17:55.406888 28367 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:17:55.409186 28382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:55.410384 28367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:17:55.410485 28367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "c231762c0bfe4a319fa0c14a1606623c"
format_stamp: "Formatted at 2026-09-29 03:17:55 on dist-test-slave-nbjg"
server_key: "a3c0f0e805a2d00fa717ad17433d09aa"
server_key_iv: "b004142a762f46847541a3249a903237"
server_key_version: "encryptionkey@0"
I20260929 03:17:55.410588 28367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:55.433645 28367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:55.436854 28367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:55.437034 28367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:55.437623 28367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:55.438472 28367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:55.438519 28367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:55.438565 28367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:55.438593 28367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:55.447285 28367 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:40529
I20260929 03:17:55.447314 28495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:40529 every 8 connection(s)
I20260929 03:17:55.448180 28367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:17:55.452375 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 28367
I20260929 03:17:55.452491 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance
I20260929 03:17:55.452788 14266 external_mini_cluster.cc:1428] Setting key 89eadac22f88fa258d3d873d69172380
Sep 29 03:17:55 dist-test-slave-nbjg krb5kdc[27992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651875, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:17:55.461553 28076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:55.449958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:42751 (local address 127.13.238.190:43239)
0929 03:17:55.450116 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:55.450120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:55.451038 (+ 918us) server_negotiation.cc:407] Connection header received
0929 03:17:55.452391 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:55.452395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:55.452462 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:55.452544 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:55.454030 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.454804 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.455954 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.456200 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.456275 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:55.458514 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:55.458517 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:55.458519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:55.458542 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:55.459811 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:55.460246 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:55.460248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:55.460250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:55.460320 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:55.460567 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:55.460569 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:55.460570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:55.460724 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:55.460795 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:55.461343 (+ 548us) server_negotiation.cc:299] Negotiation successful
0929 03:17:55.461435 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:17:55.462466 28498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:55.450409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.131:42751)
0929 03:17:55.450877 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:55.450921 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:55.452182 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:55.452751 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:55.452761 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:55.453231 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:55.453877 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:55.453901 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.454939 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.454943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:55.455857 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:55.455867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.456325 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.456328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:55.456467 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:55.456960 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:55.456975 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:55.458377 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:55.459921 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:55.459926 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:55.459938 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:55.460174 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:55.460381 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:55.460383 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:55.460384 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:55.460500 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:55.460845 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:55.460851 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:55.461112 (+ 261us) client_negotiation.cc:769] Sending connection context
0929 03:17:55.461304 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 03:17:55.461565 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":112,"threads_started":1}
I20260929 03:17:55.463598 28496 heartbeater.cc:347] Connected to a master server at 127.13.238.190:43239
I20260929 03:17:55.463830 28496 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:55.464308 28496 heartbeater.cc:511] Master 127.13.238.190:43239 requested a full tablet report, sending...
I20260929 03:17:55.465305 28037 ts_manager.cc:194] Registered new tserver with Master: c231762c0bfe4a319fa0c14a1606623c (127.13.238.131:40529)
I20260929 03:17:55.465878 28037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.131@KRBTEST.COM'} at 127.13.238.131:42751
I20260929 03:17:55.466611 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:17:55.475574 28076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:55.468773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39140 (local address 127.13.238.190:43239)
0929 03:17:55.468900 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:55.468906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:55.469122 (+ 216us) server_negotiation.cc:407] Connection header received
0929 03:17:55.469198 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:55.469201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:55.469242 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:55.469304 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:55.469972 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.470642 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.471419 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.471634 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.471717 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:55.472994 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:55.472998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:55.473000 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:55.473026 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:55.474021 (+ 995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:55.474458 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:55.474462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:55.474464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:55.474539 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:55.474773 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:55.474776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:55.474778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:55.474985 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:55.475124 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:55.475325 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 03:17:55.475427 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:17:55.478549 28036 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:17:55.481477 28036 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:17:55.495468 28510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:55.489761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:42287 (local address 127.0.0.1:46194)
0929 03:17:55.490902 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:55.490916 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:55.491007 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:55.491574 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:55.491580 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:55.491615 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:55.491861 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:55.491867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.492831 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.492834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:55.493931 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:55.493940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.494895 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.494899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:55.495045 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:55.495091 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 03:17:55.495190 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 03:17:55.495306 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1084,"mutex_wait_us":94,"spinlock_wait_cycles":8832,"thread_start_us":88,"threads_started":1}
I20260929 03:17:55.495486 28511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:55.490491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46194 (local address 127.13.238.129:42287)
0929 03:17:55.490967 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:55.490986 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:55.491010 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:17:55.491194 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:55.491201 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:55.491413 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:55.491569 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:55.491982 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.492713 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.494042 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.494778 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.494900 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:55.495031 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:55.495238 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 03:17:55.495339 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":106,"threads_started":1}
I20260929 03:17:55.495684 28507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:55.489674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:40529 (local address 127.0.0.1:54370)
0929 03:17:55.490529 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:55.490581 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:55.490699 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:55.491575 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:55.491580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:55.491614 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:55.491855 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:55.491866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.493287 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.493291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:55.494284 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:55.494295 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.495321 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.495325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:55.495409 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:55.495432 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:17:55.495478 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 03:17:55.495542 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":132,"spinlock_wait_cycles":7040,"thread_start_us":101,"threads_started":1}
I20260929 03:17:55.496255 28509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:55.489810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54370 (local address 127.13.238.131:40529)
0929 03:17:55.491205 (+ 1395us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:55.491216 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:55.491235 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:17:55.491293 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:55.491297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:55.491436 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:55.491804 (+ 368us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:55.491948 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.493137 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.494406 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.495190 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.495948 (+ 758us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:55.496043 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:55.496082 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 03:17:55.496151 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1298,"thread_start_us":130,"threads_started":1}
I20260929 03:17:55.496685 28508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:55.489934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:41871 (local address 127.0.0.1:47172)
0929 03:17:55.491038 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:55.491048 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:55.491127 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:55.491713 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:55.491716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:55.491727 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:55.491824 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:55.491831 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.493287 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.493291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:55.494997 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:55.495008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.496327 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.496331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:55.496421 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:55.496445 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 03:17:55.496506 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 03:17:55.496569 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"mutex_wait_us":56,"thread_start_us":89,"threads_started":1}
I20260929 03:17:55.497344 28512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:55.490491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47172 (local address 127.13.238.130:41871)
0929 03:17:55.490800 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:55.490812 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:55.491209 (+ 397us) server_negotiation.cc:407] Connection header received
0929 03:17:55.491277 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:55.491282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:55.491420 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:55.491579 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:55.491954 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.493137 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.495130 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.496196 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.496315 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:55.496997 (+ 682us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:55.497051 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 03:17:55.497208 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":108,"threads_started":1}
I20260929 03:17:55.498965 28430 tablet_service.cc:1467] Processing CreateTablet for tablet 249b87d855ce4464af1658ac5f34c100 (DEFAULT_TABLE table=test-table [id=c7157329ea4c41ecbebc6ed7cb1e9d99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:55.499032 28162 tablet_service.cc:1467] Processing CreateTablet for tablet 249b87d855ce4464af1658ac5f34c100 (DEFAULT_TABLE table=test-table [id=c7157329ea4c41ecbebc6ed7cb1e9d99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:55.499883 28430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 249b87d855ce4464af1658ac5f34c100. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:55.500044 28162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 249b87d855ce4464af1658ac5f34c100. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:55.499804 28296 tablet_service.cc:1467] Processing CreateTablet for tablet 249b87d855ce4464af1658ac5f34c100 (DEFAULT_TABLE table=test-table [id=c7157329ea4c41ecbebc6ed7cb1e9d99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:55.500685 28296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 249b87d855ce4464af1658ac5f34c100. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:55.506075 28513 tablet_bootstrap.cc:492] T 249b87d855ce4464af1658ac5f34c100 P c231762c0bfe4a319fa0c14a1606623c: Bootstrap starting.
I20260929 03:17:55.507455 28514 tablet_bootstrap.cc:492] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae: Bootstrap starting.
I20260929 03:17:55.508333 28513 tablet_bootstrap.cc:654] T 249b87d855ce4464af1658ac5f34c100 P c231762c0bfe4a319fa0c14a1606623c: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:55.509140 28514 tablet_bootstrap.cc:654] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:55.509413 28513 log.cc:824] T 249b87d855ce4464af1658ac5f34c100 P c231762c0bfe4a319fa0c14a1606623c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:55.510005 28515 tablet_bootstrap.cc:492] T 249b87d855ce4464af1658ac5f34c100 P c92aa75a5e8e4eb2b97286955650f3f4: Bootstrap starting.
I20260929 03:17:55.510262 28514 log.cc:824] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:55.511499 28513 tablet_bootstrap.cc:492] T 249b87d855ce4464af1658ac5f34c100 P c231762c0bfe4a319fa0c14a1606623c: No bootstrap required, opened a new log
I20260929 03:17:55.511696 28513 ts_tablet_manager.cc:1397] T 249b87d855ce4464af1658ac5f34c100 P c231762c0bfe4a319fa0c14a1606623c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 03:17:55.512125 28514 tablet_bootstrap.cc:492] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae: No bootstrap required, opened a new log
I20260929 03:17:55.512272 28515 tablet_bootstrap.cc:654] T 249b87d855ce4464af1658ac5f34c100 P c92aa75a5e8e4eb2b97286955650f3f4: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:55.512295 28514 ts_tablet_manager.cc:1397] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 03:17:55.512907 28515 log.cc:824] T 249b87d855ce4464af1658ac5f34c100 P c92aa75a5e8e4eb2b97286955650f3f4: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:55.514691 28515 tablet_bootstrap.cc:492] T 249b87d855ce4464af1658ac5f34c100 P c92aa75a5e8e4eb2b97286955650f3f4: No bootstrap required, opened a new log
I20260929 03:17:55.514866 28515 ts_tablet_manager.cc:1397] T 249b87d855ce4464af1658ac5f34c100 P c92aa75a5e8e4eb2b97286955650f3f4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 03:17:55.515610 28513 raft_consensus.cc:348] T 249b87d855ce4464af1658ac5f34c100 P c231762c0bfe4a319fa0c14a1606623c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } } peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } }
I20260929 03:17:55.515887 28513 raft_consensus.cc:374] T 249b87d855ce4464af1658ac5f34c100 P c231762c0bfe4a319fa0c14a1606623c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:55.515940 28513 raft_consensus.cc:729] T 249b87d855ce4464af1658ac5f34c100 P c231762c0bfe4a319fa0c14a1606623c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c231762c0bfe4a319fa0c14a1606623c, State: Initialized, Role: FOLLOWER
I20260929 03:17:55.516088 28514 raft_consensus.cc:348] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } } peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } }
I20260929 03:17:55.516315 28514 raft_consensus.cc:374] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:55.516364 28514 raft_consensus.cc:729] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed14d80f04c482bb017fd772a1c22ae, State: Initialized, Role: FOLLOWER
I20260929 03:17:55.516434 28513 consensus_queue.cc:260] T 249b87d855ce4464af1658ac5f34c100 P c231762c0bfe4a319fa0c14a1606623c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } } peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } }
I20260929 03:17:55.516844 28514 consensus_queue.cc:260] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } } peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } }
I20260929 03:17:55.517194 28496 heartbeater.cc:503] Master 127.13.238.190:43239 was elected leader, sending a full tablet report...
I20260929 03:17:55.517467 28513 ts_tablet_manager.cc:1428] T 249b87d855ce4464af1658ac5f34c100 P c231762c0bfe4a319fa0c14a1606623c: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260929 03:17:55.517742 28514 ts_tablet_manager.cc:1428] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 03:17:55.518023 28228 heartbeater.cc:503] Master 127.13.238.190:43239 was elected leader, sending a full tablet report...
I20260929 03:17:55.518484 28515 raft_consensus.cc:348] T 249b87d855ce4464af1658ac5f34c100 P c92aa75a5e8e4eb2b97286955650f3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } } peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } }
I20260929 03:17:55.518920 28515 raft_consensus.cc:374] T 249b87d855ce4464af1658ac5f34c100 P c92aa75a5e8e4eb2b97286955650f3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:55.518985 28515 raft_consensus.cc:729] T 249b87d855ce4464af1658ac5f34c100 P c92aa75a5e8e4eb2b97286955650f3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92aa75a5e8e4eb2b97286955650f3f4, State: Initialized, Role: FOLLOWER
I20260929 03:17:55.519409 28515 consensus_queue.cc:260] T 249b87d855ce4464af1658ac5f34c100 P c92aa75a5e8e4eb2b97286955650f3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } } peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } }
I20260929 03:17:55.520051 28362 heartbeater.cc:503] Master 127.13.238.190:43239 was elected leader, sending a full tablet report...
I20260929 03:17:55.521255 28515 ts_tablet_manager.cc:1428] T 249b87d855ce4464af1658ac5f34c100 P c92aa75a5e8e4eb2b97286955650f3f4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 03:17:55.599323 28520 raft_consensus.cc:482] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:17:55.599551 28520 raft_consensus.cc:504] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } } peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } }
I20260929 03:17:55.600734 28520 leader_election.cc:288] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c231762c0bfe4a319fa0c14a1606623c (127.13.238.131:40529), c92aa75a5e8e4eb2b97286955650f3f4 (127.13.238.130:41871)
I20260929 03:17:55.605945 28512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:55.601174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:44143 (local address 127.13.238.130:41871)
0929 03:17:55.601297 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:55.601301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:55.601480 (+ 179us) server_negotiation.cc:407] Connection header received
0929 03:17:55.601643 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:55.601646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:55.601698 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:55.601775 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:55.602111 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.602921 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.604678 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.605326 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.605451 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:55.605506 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:55.605731 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 03:17:55.605807 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:17:55.605945 28509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:55.601064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:50311 (local address 127.13.238.131:40529)
0929 03:17:55.601235 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:55.601240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:55.601582 (+ 342us) server_negotiation.cc:407] Connection header received
0929 03:17:55.601641 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:55.601644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:55.601697 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:55.601778 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:55.602111 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.602936 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.604678 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.605326 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.605450 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:55.605506 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:55.605731 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 03:17:55.605807 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 03:17:55.605998 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:55.600991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:40529 (local address 127.13.238.129:50311)
0929 03:17:55.601489 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:55.601501 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:55.601575 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:55.601796 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:55.601799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:55.601824 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:55.602008 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:55.602016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.603034 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.603037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:55.604534 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:55.604547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.605451 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.605456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:55.605607 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:55.605638 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 03:17:55.605709 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 03:17:55.605818 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":276,"spinlock_wait_cycles":1408,"thread_start_us":87,"threads_started":1}
I20260929 03:17:55.606392 28523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:55.600988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:41871 (local address 127.13.238.129:44143)
0929 03:17:55.601385 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:55.601398 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:55.601525 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:55.601796 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:55.601799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:55.601829 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:55.602009 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:55.602016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.603034 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.603037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:55.604534 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:55.604547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.605451 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.605456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:55.605601 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:55.605630 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 03:17:55.605709 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 03:17:55.605815 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":143,"spinlock_wait_cycles":7424,"thread_start_us":166,"threads_started":1}
I20260929 03:17:55.606652 28450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "249b87d855ce4464af1658ac5f34c100" candidate_uuid: "7ed14d80f04c482bb017fd772a1c22ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c231762c0bfe4a319fa0c14a1606623c" is_pre_election: true
I20260929 03:17:55.607026 28450 raft_consensus.cc:2454] T 249b87d855ce4464af1658ac5f34c100 P c231762c0bfe4a319fa0c14a1606623c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ed14d80f04c482bb017fd772a1c22ae in term 0.
I20260929 03:17:55.607057 28316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "249b87d855ce4464af1658ac5f34c100" candidate_uuid: "7ed14d80f04c482bb017fd772a1c22ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" is_pre_election: true
I20260929 03:17:55.607357 28316 raft_consensus.cc:2454] T 249b87d855ce4464af1658ac5f34c100 P c92aa75a5e8e4eb2b97286955650f3f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ed14d80f04c482bb017fd772a1c22ae in term 0.
I20260929 03:17:55.607606 28115 leader_election.cc:302] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ed14d80f04c482bb017fd772a1c22ae, c231762c0bfe4a319fa0c14a1606623c; no voters:
I20260929 03:17:55.607906 28520 raft_consensus.cc:2781] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:17:55.607983 28520 raft_consensus.cc:482] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:17:55.608033 28520 raft_consensus.cc:3037] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:55.609076 28520 raft_consensus.cc:504] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } } peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } }
I20260929 03:17:55.609395 28520 leader_election.cc:288] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae [CANDIDATE]: Term 1 election: Requested vote from peers c231762c0bfe4a319fa0c14a1606623c (127.13.238.131:40529), c92aa75a5e8e4eb2b97286955650f3f4 (127.13.238.130:41871)
I20260929 03:17:55.609748 28316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "249b87d855ce4464af1658ac5f34c100" candidate_uuid: "7ed14d80f04c482bb017fd772a1c22ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92aa75a5e8e4eb2b97286955650f3f4"
I20260929 03:17:55.609860 28316 raft_consensus.cc:3037] T 249b87d855ce4464af1658ac5f34c100 P c92aa75a5e8e4eb2b97286955650f3f4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:55.611042 28316 raft_consensus.cc:2454] T 249b87d855ce4464af1658ac5f34c100 P c92aa75a5e8e4eb2b97286955650f3f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ed14d80f04c482bb017fd772a1c22ae in term 1.
I20260929 03:17:55.611286 28450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "249b87d855ce4464af1658ac5f34c100" candidate_uuid: "7ed14d80f04c482bb017fd772a1c22ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c231762c0bfe4a319fa0c14a1606623c"
I20260929 03:17:55.611323 28116 leader_election.cc:302] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ed14d80f04c482bb017fd772a1c22ae, c92aa75a5e8e4eb2b97286955650f3f4; no voters:
I20260929 03:17:55.611402 28450 raft_consensus.cc:3037] T 249b87d855ce4464af1658ac5f34c100 P c231762c0bfe4a319fa0c14a1606623c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:55.611466 28520 raft_consensus.cc:2781] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:55.611685 28520 raft_consensus.cc:686] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae [term 1 LEADER]: Becoming Leader. State: Replica: 7ed14d80f04c482bb017fd772a1c22ae, State: Running, Role: LEADER
I20260929 03:17:55.612036 28520 consensus_queue.cc:237] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } } peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } }
I20260929 03:17:55.612504 28450 raft_consensus.cc:2454] T 249b87d855ce4464af1658ac5f34c100 P c231762c0bfe4a319fa0c14a1606623c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ed14d80f04c482bb017fd772a1c22ae in term 1.
I20260929 03:17:55.615159 28035 catalog_manager.cc:5495] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ed14d80f04c482bb017fd772a1c22ae (127.13.238.129). New cstate: current_term: 1 leader_uuid: "7ed14d80f04c482bb017fd772a1c22ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } health_report { overall_health: UNKNOWN } } }
I20260929 03:17:55.648371 28511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:55.644887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46198 (local address 127.13.238.129:42287)
0929 03:17:55.645003 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:55.645006 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:55.645018 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:17:55.645098 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:55.645100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:55.645158 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:55.645248 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:17:55.645482 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.646038 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.646910 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:55.647141 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:55.647226 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:55.647501 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:17:55.647942 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:17:55.648042 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:55.648200 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 03:17:55.648256 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:17:55.654536 28316 raft_consensus.cc:1261] T 249b87d855ce4464af1658ac5f34c100 P c92aa75a5e8e4eb2b97286955650f3f4 [term 1 FOLLOWER]: Refusing update from remote peer 7ed14d80f04c482bb017fd772a1c22ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:55.654536 28450 raft_consensus.cc:1261] T 249b87d855ce4464af1658ac5f34c100 P c231762c0bfe4a319fa0c14a1606623c [term 1 FOLLOWER]: Refusing update from remote peer 7ed14d80f04c482bb017fd772a1c22ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:55.655191 28520 consensus_queue.cc:1035] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:55.655385 28524 consensus_queue.cc:1035] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:55.662060 28527 mvcc.cc:205] Tried to move back new op lower bound from 7334510082673491968 to 7334510082509791232. Current Snapshot: MvccSnapshot[applied={T|T < 7334510082673491968}]
I20260929 03:17:55.663830 28529 mvcc.cc:205] Tried to move back new op lower bound from 7334510082673491968 to 7334510082509791232. Current Snapshot: MvccSnapshot[applied={T|T < 7334510082673491968}]
I20260929 03:17:55.671032 28035 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:17:55.671236 28035 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:17:55.673529 28035 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d: Sending DeleteTablet for 3 replicas of tablet 249b87d855ce4464af1658ac5f34c100
I20260929 03:17:55.674293 28430 tablet_service.cc:1514] Processing DeleteTablet for tablet 249b87d855ce4464af1658ac5f34c100 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:55 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:54370
I20260929 03:17:55.674290 28162 tablet_service.cc:1514] Processing DeleteTablet for tablet 249b87d855ce4464af1658ac5f34c100 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:55 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:46194
I20260929 03:17:55.674316 28296 tablet_service.cc:1514] Processing DeleteTablet for tablet 249b87d855ce4464af1658ac5f34c100 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:55 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:47172
I20260929 03:17:55.674803 28536 tablet_replica.cc:330] T 249b87d855ce4464af1658ac5f34c100 P c92aa75a5e8e4eb2b97286955650f3f4: stopping tablet replica
I20260929 03:17:55.675128 28536 raft_consensus.cc:2229] T 249b87d855ce4464af1658ac5f34c100 P c92aa75a5e8e4eb2b97286955650f3f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:17:55.675231 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 28099
I20260929 03:17:55.675468 28536 raft_consensus.cc:2258] T 249b87d855ce4464af1658ac5f34c100 P c92aa75a5e8e4eb2b97286955650f3f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:55.675873 28538 tablet_replica.cc:330] T 249b87d855ce4464af1658ac5f34c100 P c231762c0bfe4a319fa0c14a1606623c: stopping tablet replica
I20260929 03:17:55.676165 28538 raft_consensus.cc:2229] T 249b87d855ce4464af1658ac5f34c100 P c231762c0bfe4a319fa0c14a1606623c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:17:55.676433 28538 raft_consensus.cc:2258] T 249b87d855ce4464af1658ac5f34c100 P c231762c0bfe4a319fa0c14a1606623c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:55.677850 28538 ts_tablet_manager.cc:1905] T 249b87d855ce4464af1658ac5f34c100 P c231762c0bfe4a319fa0c14a1606623c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:55.678032 28536 ts_tablet_manager.cc:1905] T 249b87d855ce4464af1658ac5f34c100 P c92aa75a5e8e4eb2b97286955650f3f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:55.680117 28538 ts_tablet_manager.cc:1918] T 249b87d855ce4464af1658ac5f34c100 P c231762c0bfe4a319fa0c14a1606623c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:17:55.680204 28538 log.cc:1196] T 249b87d855ce4464af1658ac5f34c100 P c231762c0bfe4a319fa0c14a1606623c: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/249b87d855ce4464af1658ac5f34c100
I20260929 03:17:55.680474 28538 ts_tablet_manager.cc:1939] T 249b87d855ce4464af1658ac5f34c100 P c231762c0bfe4a319fa0c14a1606623c: Deleting consensus metadata
I20260929 03:17:55.680600 28536 ts_tablet_manager.cc:1918] T 249b87d855ce4464af1658ac5f34c100 P c92aa75a5e8e4eb2b97286955650f3f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:17:55.680706 28536 log.cc:1196] T 249b87d855ce4464af1658ac5f34c100 P c92aa75a5e8e4eb2b97286955650f3f4: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/wal/wals/249b87d855ce4464af1658ac5f34c100
I20260929 03:17:55.680963 28536 ts_tablet_manager.cc:1939] T 249b87d855ce4464af1658ac5f34c100 P c92aa75a5e8e4eb2b97286955650f3f4: Deleting consensus metadata
I20260929 03:17:55.681659 28021 catalog_manager.cc:4841] TS c231762c0bfe4a319fa0c14a1606623c (127.13.238.131:40529): tablet 249b87d855ce4464af1658ac5f34c100 (table test-table [id=c7157329ea4c41ecbebc6ed7cb1e9d99]) successfully deleted
I20260929 03:17:55.682179 28022 catalog_manager.cc:4841] TS c92aa75a5e8e4eb2b97286955650f3f4 (127.13.238.130:41871): tablet 249b87d855ce4464af1658ac5f34c100 (table test-table [id=c7157329ea4c41ecbebc6ed7cb1e9d99]) successfully deleted
I20260929 03:17:55.683576 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 28233
W20260929 03:17:55.683738 28024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.238.129:42287 (error 108)
I20260929 03:17:55.689332 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 28367
I20260929 03:17:55.690048 28508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:55.689674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:42287 (local address 127.0.0.1:46208)
0929 03:17:55.689889 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:55.689957 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.129:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260929 03:17:55.690366 28024 catalog_manager.cc:4584] TS 7ed14d80f04c482bb017fd772a1c22ae (127.13.238.129:42287): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 249b87d855ce4464af1658ac5f34c100: Network error: Client connection negotiation failed: client connection to 127.13.238.129:42287: connect: Connection refused (error 111)
I20260929 03:17:55.695089 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 28005
I20260929 03:17:55.700594 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:43239
--webserver_interface=127.13.238.190
--webserver_port=43197
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:43239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:55.812124 28540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:55.812397 28540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:55.812457 28540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:55.816452 28540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:55.816505 28540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:55.816530 28540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:55.816552 28540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:55.816572 28540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:17:55.821826 28540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:43239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:43239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=43197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:17:55.822897 28540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:55.823906 28540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:55.838511 28547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:55.838510 28546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:55.838510 28549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:55.838564 28540 server_base.cc:1034] running on GCE node
I20260929 03:17:55.839419 28540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:55.840582 28540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:55.841746 28540 hybrid_clock.cc:648] HybridClock initialized: now 1790651875841714 us; error 46 us; skew 500 ppm
Sep 29 03:17:55 dist-test-slave-nbjg krb5kdc[27992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651875, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:55.844594 28540 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:17:55.845546 28540 webserver.cc:460] Webserver started at http://127.13.238.190:43197/ using document root <none> and password file <none>
I20260929 03:17:55.846190 28540 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:55.846309 28540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:55.850567 28540 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:17:55.852869 28556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:55.854152 28540 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:17:55.854317 28540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "678678b2d3d24b57b34c646855d8b52d"
format_stamp: "Formatted at 2026-09-29 03:17:54 on dist-test-slave-nbjg"
server_key: "801aa173609a1d676c827bde99fe34b8"
server_key_iv: "35be0706771725e6669041506764235e"
server_key_version: "encryptionkey@0"
I20260929 03:17:55.854946 28540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:55.869593 28540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:55.872548 28540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:55.872736 28540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:55.879671 28540 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:43239
I20260929 03:17:55.879935 28608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:43239 every 8 connection(s)
I20260929 03:17:55.880775 28540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:17:55.885301 28609 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:17:55.886924 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 28540
I20260929 03:17:55.886994 28609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d: Bootstrap starting.
I20260929 03:17:55.887387 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:42287
--local_ip_for_outbound_sockets=127.13.238.129
--tserver_master_addrs=127.13.238.190:43239
--webserver_port=45851
--webserver_interface=127.13.238.129
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:17:55.903626 28609 log.cc:824] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:55.912412 28609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:17:55.912760 28609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d: Bootstrap complete.
I20260929 03:17:55.916615 28609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678678b2d3d24b57b34c646855d8b52d" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43239 } }
I20260929 03:17:55.917028 28609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 678678b2d3d24b57b34c646855d8b52d, State: Initialized, Role: FOLLOWER
I20260929 03:17:55.917531 28609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678678b2d3d24b57b34c646855d8b52d" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43239 } }
I20260929 03:17:55.917645 28609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:17:55.917711 28609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:17:55.917798 28609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:17:55.919692 28609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678678b2d3d24b57b34c646855d8b52d" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43239 } }
I20260929 03:17:55.920006 28609 leader_election.cc:302] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 678678b2d3d24b57b34c646855d8b52d; no voters:
I20260929 03:17:55.920300 28609 leader_election.cc:288] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 03:17:55.920401 28613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:17:55.920611 28613 raft_consensus.cc:686] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [term 2 LEADER]: Becoming Leader. State: Replica: 678678b2d3d24b57b34c646855d8b52d, State: Running, Role: LEADER
I20260929 03:17:55.920900 28613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678678b2d3d24b57b34c646855d8b52d" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43239 } }
I20260929 03:17:55.921574 28609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:55.921830 28613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 678678b2d3d24b57b34c646855d8b52d. Latest consensus state: current_term: 2 leader_uuid: "678678b2d3d24b57b34c646855d8b52d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678678b2d3d24b57b34c646855d8b52d" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43239 } } }
I20260929 03:17:55.921949 28613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:55.922317 28617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:17:55.922403 28614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "678678b2d3d24b57b34c646855d8b52d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678678b2d3d24b57b34c646855d8b52d" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43239 } } }
I20260929 03:17:55.922595 28614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:55.927055 28617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:17:55.927666 28617 catalog_manager.cc:1259] Loaded cluster ID: 6655e4cd99034f35a1cc590422e33d7e
I20260929 03:17:55.927752 28617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:17:55.928987 28617 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 03:17:55.931761 28617 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 03:17:55.935868 28617 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:17:55.936546 28617 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d: Loaded TSK: 0
I20260929 03:17:55.937485 28617 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:17:56.007927 28611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:56.008212 28611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:56.008273 28611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:56.012882 28611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:56.012943 28611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:56.013002 28611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:17:56.018332 28611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:42287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=45851
--tserver_master_addrs=127.13.238.190:43239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:17:56.019332 28611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:56.020319 28611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:56.030566 28637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:56.030560 28636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:56.030824 28611 server_base.cc:1034] running on GCE node
W20260929 03:17:56.031030 28639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:56.031467 28611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:56.032318 28611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:56.033531 28611 hybrid_clock.cc:648] HybridClock initialized: now 1790651876033490 us; error 45 us; skew 500 ppm
Sep 29 03:17:56 dist-test-slave-nbjg krb5kdc[27992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651876, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:56.037165 28611 init.cc:377] Logged in from keytab as kudu/127.13.238.129@KRBTEST.COM (short username kudu)
I20260929 03:17:56.038102 28611 webserver.cc:460] Webserver started at http://127.13.238.129:45851/ using document root <none> and password file <none>
I20260929 03:17:56.038683 28611 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:56.038755 28611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:56.042749 28611 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:17:56.045069 28646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:56.046444 28611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:17:56.046555 28611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "7ed14d80f04c482bb017fd772a1c22ae"
format_stamp: "Formatted at 2026-09-29 03:17:54 on dist-test-slave-nbjg"
server_key: "0203449a79a66624da04a0c128d8c1bf"
server_key_iv: "421357bffe42a88dedb6260c2ab2491f"
server_key_version: "encryptionkey@0"
I20260929 03:17:56.047124 28611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:56.062069 28611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:56.065157 28611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:56.065330 28611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:56.065905 28611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:56.067051 28653 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 03:17:56.072034 28611 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 03:17:56.072114 28611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260929 03:17:56.072181 28611 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 03:17:56.073872 28611 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 03:17:56.073917 28611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 03:17:56.074293 28653 tablet_bootstrap.cc:492] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae: Bootstrap starting.
I20260929 03:17:56.083189 28611 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:42287
I20260929 03:17:56.083283 28760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:42287 every 8 connection(s)
I20260929 03:17:56.084167 28611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:17:56.092446 28653 log.cc:824] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:56.093895 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 28611
I20260929 03:17:56.094405 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:41871
--local_ip_for_outbound_sockets=127.13.238.130
--tserver_master_addrs=127.13.238.190:43239
--webserver_port=45469
--webserver_interface=127.13.238.130
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:17:56 dist-test-slave-nbjg krb5kdc[27992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651876, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:17:56.101284 28653 tablet_bootstrap.cc:492] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:17:56.101703 28653 tablet_bootstrap.cc:492] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae: Bootstrap complete.
I20260929 03:17:56.102867 28764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:56.086119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:55129 (local address 127.13.238.190:43239)
0929 03:17:56.086600 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:56.086619 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:56.086738 (+ 119us) server_negotiation.cc:407] Connection header received
0929 03:17:56.088182 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:56.088194 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:56.088490 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:56.088756 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:56.090376 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.091358 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.092958 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.093665 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.093928 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:56.096852 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:56.096857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:56.096871 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:56.096904 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:56.099011 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:56.099672 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:56.099678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:56.099684 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:56.099841 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:56.100241 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:56.100245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:56.100247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:56.100724 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:56.100857 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:56.101694 (+ 837us) server_negotiation.cc:299] Negotiation successful
0929 03:17:56.102009 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":127,"threads_started":1}
I20260929 03:17:56.102977 28653 ts_tablet_manager.cc:1397] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.015s
I20260929 03:17:56.103237 28763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:56.085975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.129:55129)
0929 03:17:56.086578 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:56.086636 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:56.087745 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:56.088941 (+ 1196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:56.088954 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:56.089439 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:56.090136 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:56.090160 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.091528 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.091532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:56.092820 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:56.092833 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.093859 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.093877 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:56.094096 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:56.094793 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:56.094813 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:56.096624 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:56.099145 (+ 2521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:56.099152 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:56.099168 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:56.099531 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:56.099939 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:56.099942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:56.099944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:56.100124 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:56.100845 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:56.100851 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:56.101162 (+ 311us) client_negotiation.cc:769] Sending connection context
0929 03:17:56.101495 (+ 333us) client_negotiation.cc:240] Negotiation successful
0929 03:17:56.101910 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":174,"threads_started":1}
I20260929 03:17:56.105839 28761 heartbeater.cc:347] Connected to a master server at 127.13.238.190:43239
I20260929 03:17:56.105566 28653 raft_consensus.cc:348] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } } peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } }
I20260929 03:17:56.106034 28653 raft_consensus.cc:729] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed14d80f04c482bb017fd772a1c22ae, State: Initialized, Role: FOLLOWER
I20260929 03:17:56.106077 28761 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:56.106529 28653 consensus_queue.cc:260] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } } peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } }
I20260929 03:17:56.106644 28761 heartbeater.cc:511] Master 127.13.238.190:43239 requested a full tablet report, sending...
I20260929 03:17:56.107522 28653 ts_tablet_manager.cc:1428] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260929 03:17:56.109304 28573 ts_manager.cc:194] Registered new tserver with Master: 7ed14d80f04c482bb017fd772a1c22ae (127.13.238.129:42287)
I20260929 03:17:56.113621 28573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.129@KRBTEST.COM'} at 127.13.238.129:55129
I20260929 03:17:56.116250 28761 heartbeater.cc:503] Master 127.13.238.190:43239 was elected leader, sending a full tablet report...
Sep 29 03:17:56 dist-test-slave-nbjg krb5kdc[27992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651875, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for kudu/127.13.238.129@KRBTEST.COM
I20260929 03:17:56.123062 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:56.112567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:42287 (local address 127.0.0.1:33642)
0929 03:17:56.112949 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:56.113061 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:56.113189 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:56.113813 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:56.113819 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:56.114164 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:56.114309 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:56.114317 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.115586 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.115590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:56.116565 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:56.116579 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.117232 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.117236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:56.117354 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:56.117985 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:56.118002 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:56.119560 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:56.121267 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:56.121276 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:56.121281 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:56.121579 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:56.121827 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:56.121830 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:56.121832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:56.121924 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:56.122451 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:56.122456 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:56.122578 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 03:17:56.122711 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 03:17:56.122919 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":100,"threads_started":1}
I20260929 03:17:56.123134 28769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:56.112643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33642 (local address 127.13.238.129:42287)
0929 03:17:56.113327 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:56.113341 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:56.113360 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:17:56.113411 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:56.113414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:56.113588 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:56.113704 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:56.114445 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.115457 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.116681 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.117105 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.117203 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:56.119706 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:56.119715 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:56.119726 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:56.119749 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:56.121160 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:56.121672 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:56.121677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:56.121678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:56.121751 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:56.122032 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:56.122036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:56.122037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:56.122346 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:56.122483 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:56.122809 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 03:17:56.122979 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":97,"threads_started":1}
I20260929 03:17:56.124346 28695 tablet_service.cc:1514] Processing DeleteTablet for tablet 249b87d855ce4464af1658ac5f34c100 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:55 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:33642
I20260929 03:17:56.125160 28770 tablet_replica.cc:330] stopping tablet replica
I20260929 03:17:56.125428 28770 raft_consensus.cc:2229] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:17:56.125654 28770 raft_consensus.cc:2258] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:56.126605 28770 ts_tablet_manager.cc:1905] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:56.130119 28770 ts_tablet_manager.cc:1918] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:17:56.130223 28770 log.cc:1196] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/wal/wals/249b87d855ce4464af1658ac5f34c100
I20260929 03:17:56.130456 28770 ts_tablet_manager.cc:1939] T 249b87d855ce4464af1658ac5f34c100 P 7ed14d80f04c482bb017fd772a1c22ae: Deleting consensus metadata
I20260929 03:17:56.132097 28560 catalog_manager.cc:4841] TS 7ed14d80f04c482bb017fd772a1c22ae (127.13.238.129:42287): tablet 249b87d855ce4464af1658ac5f34c100 (table test-table [id=c7157329ea4c41ecbebc6ed7cb1e9d99]) successfully deleted
W20260929 03:17:56.211534 28766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:56.211802 28766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:56.211859 28766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:56.215844 28766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:56.215907 28766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:56.215993 28766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:17:56.221302 28766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:41871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=45469
--tserver_master_addrs=127.13.238.190:43239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:17:56.222411 28766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:56.223471 28766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:17:56.233057 28766 server_base.cc:1034] running on GCE node
W20260929 03:17:56.233000 28775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:56.233017 28778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:56.233071 28776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:56.233635 28766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:56.234299 28766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:56.235463 28766 hybrid_clock.cc:648] HybridClock initialized: now 1790651876235441 us; error 34 us; skew 500 ppm
Sep 29 03:17:56 dist-test-slave-nbjg krb5kdc[27992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651876, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:56.238250 28766 init.cc:377] Logged in from keytab as kudu/127.13.238.130@KRBTEST.COM (short username kudu)
I20260929 03:17:56.239115 28766 webserver.cc:460] Webserver started at http://127.13.238.130:45469/ using document root <none> and password file <none>
I20260929 03:17:56.239661 28766 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:56.239748 28766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:56.243731 28766 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:17:56.245716 28785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:56.246915 28766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:17:56.247023 28766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "c92aa75a5e8e4eb2b97286955650f3f4"
format_stamp: "Formatted at 2026-09-29 03:17:55 on dist-test-slave-nbjg"
server_key: "6e0af4329db494a53b431ed7f300d266"
server_key_iv: "ed02205b4b588768e83f79a1a452a9c7"
server_key_version: "encryptionkey@0"
I20260929 03:17:56.247591 28766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:56.262059 28766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:56.265178 28766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:56.265349 28766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:56.265930 28766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:56.266812 28766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:56.266860 28766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:56.266913 28766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:56.266947 28766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:56.275313 28766 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:41871
I20260929 03:17:56.275350 28898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:41871 every 8 connection(s)
I20260929 03:17:56.276310 28766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:17:56.281502 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 28766
I20260929 03:17:56.281936 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:40529
--local_ip_for_outbound_sockets=127.13.238.131
--tserver_master_addrs=127.13.238.190:43239
--webserver_port=39617
--webserver_interface=127.13.238.131
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:17:56 dist-test-slave-nbjg krb5kdc[27992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651876, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:17:56.290911 28764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:56.278156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:54741 (local address 127.13.238.190:43239)
0929 03:17:56.278327 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:56.278332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:56.279242 (+ 910us) server_negotiation.cc:407] Connection header received
0929 03:17:56.280562 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:56.280566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:56.280626 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:56.280700 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:56.282141 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.282896 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.284201 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.284455 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.284562 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:56.287128 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:56.287132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:56.287136 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:56.287168 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:56.288668 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:56.289187 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:56.289191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:56.289193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:56.289283 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:56.289653 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:56.289656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:56.289658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:56.289887 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:56.289988 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:56.290593 (+ 605us) server_negotiation.cc:299] Negotiation successful
0929 03:17:56.290734 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 03:17:56.291744 28901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:56.278596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.130:54741)
0929 03:17:56.279100 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:56.279142 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:56.280352 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:56.280909 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:56.280916 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:56.281288 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:56.281952 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:56.281986 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.283061 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.283064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:56.284085 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:56.284095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.284626 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.284629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:56.284777 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:56.285306 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:56.285322 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:56.286930 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:56.288798 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:56.288803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:56.288817 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:56.289088 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:56.289398 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:56.289401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:56.289402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:56.289559 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:56.290040 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:56.290046 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:56.290349 (+ 303us) client_negotiation.cc:769] Sending connection context
0929 03:17:56.290534 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 03:17:56.290802 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":126,"threads_started":1}
I20260929 03:17:56.293089 28899 heartbeater.cc:347] Connected to a master server at 127.13.238.190:43239
I20260929 03:17:56.293331 28899 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:56.293829 28899 heartbeater.cc:511] Master 127.13.238.190:43239 requested a full tablet report, sending...
I20260929 03:17:56.294858 28573 ts_manager.cc:194] Registered new tserver with Master: c92aa75a5e8e4eb2b97286955650f3f4 (127.13.238.130:41871)
I20260929 03:17:56.295608 28573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.130@KRBTEST.COM'} at 127.13.238.130:54741
W20260929 03:17:56.391825 28902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:56.392098 28902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:56.392140 28902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:56.396166 28902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:56.396232 28902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:56.396291 28902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:17:56.401481 28902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:40529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=39617
--tserver_master_addrs=127.13.238.190:43239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:17:56.402535 28902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:56.403503 28902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:56.412865 28910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:56.412878 28907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:56.412885 28908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:56.412964 28902 server_base.cc:1034] running on GCE node
I20260929 03:17:56.413590 28902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:56.414275 28902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:56.415463 28902 hybrid_clock.cc:648] HybridClock initialized: now 1790651876415430 us; error 50 us; skew 500 ppm
Sep 29 03:17:56 dist-test-slave-nbjg krb5kdc[27992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651876, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:56.418241 28902 init.cc:377] Logged in from keytab as kudu/127.13.238.131@KRBTEST.COM (short username kudu)
I20260929 03:17:56.419138 28902 webserver.cc:460] Webserver started at http://127.13.238.131:39617/ using document root <none> and password file <none>
I20260929 03:17:56.419691 28902 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:56.419771 28902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:56.423774 28902 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:17:56.425796 28917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:56.427109 28902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:17:56.427229 28902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "c231762c0bfe4a319fa0c14a1606623c"
format_stamp: "Formatted at 2026-09-29 03:17:55 on dist-test-slave-nbjg"
server_key: "a3c0f0e805a2d00fa717ad17433d09aa"
server_key_iv: "b004142a762f46847541a3249a903237"
server_key_version: "encryptionkey@0"
I20260929 03:17:56.427779 28902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:56.449249 28902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:56.452643 28902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:56.452816 28902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:56.453425 28902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:56.454305 28902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:56.454355 28902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:56.454409 28902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:56.454433 28902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:56.462918 28902 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:40529
I20260929 03:17:56.462954 29030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:40529 every 8 connection(s)
I20260929 03:17:56.463881 28902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:17:56.468619 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 28902
Sep 29 03:17:56 dist-test-slave-nbjg krb5kdc[27992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651876, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:17:56.477434 29035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:56.469348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46600 (local address 127.13.238.190:43239)
0929 03:17:56.469861 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:56.469865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:56.469877 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:17:56.469908 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:56.469910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:56.469943 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:56.470014 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:56.470854 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.471346 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.472337 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.472554 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.472640 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:56.473885 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:56.473890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:56.473892 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:56.473920 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:56.475749 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:56.476176 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:56.476179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:56.476180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:56.476256 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:56.476529 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:56.476531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:56.476533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:56.476731 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:56.476870 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:56.477096 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 03:17:56.477265 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":82,"threads_started":1}
I20260929 03:17:56.477998 28764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:56.465673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:60761 (local address 127.13.238.190:43239)
0929 03:17:56.465805 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:56.465809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:56.466688 (+ 879us) server_negotiation.cc:407] Connection header received
0929 03:17:56.468067 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:56.468070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:56.468120 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:56.468193 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:56.469763 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.470315 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.471642 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.471853 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.471926 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:56.474475 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:56.474478 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:56.474481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:56.474522 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:56.475981 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:56.476498 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:56.476501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:56.476503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:56.476580 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:56.476947 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:56.476950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:56.476951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:56.477123 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:56.477215 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:56.477762 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 03:17:56.477865 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:17:56.478971 29033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:56.466041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.131:60761)
0929 03:17:56.466543 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:56.466586 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:56.467866 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:56.468444 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:56.468455 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:56.468965 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:56.469596 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:56.469617 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.470432 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.470436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:56.471522 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:56.471533 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.471989 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.471993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:56.472187 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:56.472769 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:56.472785 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:56.474308 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:56.476093 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:56.476098 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:56.476114 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:56.476408 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:56.476684 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:56.476686 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:56.476687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:56.476842 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:56.477236 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:56.477243 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:56.477501 (+ 258us) client_negotiation.cc:769] Sending connection context
0929 03:17:56.477709 (+ 208us) client_negotiation.cc:240] Negotiation successful
0929 03:17:56.478032 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":153,"threads_started":1}
I20260929 03:17:56.479955 29031 heartbeater.cc:347] Connected to a master server at 127.13.238.190:43239
I20260929 03:17:56.480180 29031 heartbeater.cc:464] Registering TS with master...
I20260929 03:17:56.480639 29031 heartbeater.cc:511] Master 127.13.238.190:43239 requested a full tablet report, sending...
I20260929 03:17:56.481555 28573 ts_manager.cc:194] Registered new tserver with Master: c231762c0bfe4a319fa0c14a1606623c (127.13.238.131:40529)
I20260929 03:17:56.482087 28573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.131@KRBTEST.COM'} at 127.13.238.131:60761
I20260929 03:17:56.490834 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:17:56.494622 28764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:56.491398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46616 (local address 127.13.238.190:43239)
0929 03:17:56.491501 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:56.491504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:56.491697 (+ 193us) server_negotiation.cc:407] Connection header received
0929 03:17:56.491824 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:56.491827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:56.491917 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:56.491999 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:17:56.492162 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.492643 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.493406 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.493615 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.493685 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:56.493906 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:17:56.494359 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:17:56.494446 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:56.494487 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 03:17:56.494529 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:17:56.495222 28573 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:46616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:17:56.498418 28573 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:17:56.505774 28695 tablet_service.cc:1467] Processing CreateTablet for tablet a04c8e8cdb19420bb3405b252166ce26 (DEFAULT_TABLE table=test-table [id=ec87b5e0120f4e4084adb9cee0a72369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:56.506479 28695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a04c8e8cdb19420bb3405b252166ce26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:56.509322 28653 tablet_bootstrap.cc:492] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae: Bootstrap starting.
I20260929 03:17:56.509903 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:56.504740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:41871 (local address 127.0.0.1:56976)
0929 03:17:56.504845 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:56.504860 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:56.504934 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:56.505748 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:56.505751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:56.505769 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:56.505849 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:56.505854 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.507273 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.507276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:56.508480 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:56.508491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.509556 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.509560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:56.509657 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:56.509701 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 03:17:56.509743 (+ 42us) client_negotiation.cc:240] Negotiation successful
0929 03:17:56.509803 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":22}
I20260929 03:17:56.510063 29041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:56.505201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:40529 (local address 127.0.0.1:55048)
0929 03:17:56.505484 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:56.505496 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:56.505578 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:56.506264 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:56.506267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:56.506285 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:56.506365 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:56.506371 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.507586 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.507589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:56.508693 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:56.508701 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.509641 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.509644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:56.509740 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:56.509765 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 03:17:56.509813 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 03:17:56.509877 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":38,"thread_start_us":87,"threads_started":1}
I20260929 03:17:56.510249 28653 tablet_bootstrap.cc:654] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:56.510819 29042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:56.505454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55048 (local address 127.13.238.131:40529)
0929 03:17:56.505819 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:56.505829 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:56.505850 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:17:56.505921 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:56.505926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:56.506105 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:56.506296 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:56.506463 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.507447 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.508799 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.509499 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.510391 (+ 892us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:56.510541 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:56.510597 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 03:17:56.510689 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":136,"threads_started":1}
I20260929 03:17:56.510887 29040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:56.504930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56976 (local address 127.13.238.130:41871)
0929 03:17:56.505238 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:56.505249 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:56.505274 (+ 25us) server_negotiation.cc:407] Connection header received
0929 03:17:56.505352 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:56.505357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:56.505582 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:56.505980 (+ 398us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:56.506135 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.507152 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.508622 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.509433 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.510090 (+ 657us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:56.510492 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:56.510543 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 03:17:56.510633 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":126,"threads_started":1}
I20260929 03:17:56.511736 28653 tablet_bootstrap.cc:492] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae: No bootstrap required, opened a new log
I20260929 03:17:56.511819 28653 ts_tablet_manager.cc:1397] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:17:56.512241 28653 raft_consensus.cc:348] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } } peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } }
I20260929 03:17:56.512346 28653 raft_consensus.cc:374] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:56.512379 28653 raft_consensus.cc:729] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed14d80f04c482bb017fd772a1c22ae, State: Initialized, Role: FOLLOWER
I20260929 03:17:56.512498 28653 consensus_queue.cc:260] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } } peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } }
I20260929 03:17:56.512753 28653 ts_tablet_manager.cc:1428] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 03:17:56.513459 28965 tablet_service.cc:1467] Processing CreateTablet for tablet a04c8e8cdb19420bb3405b252166ce26 (DEFAULT_TABLE table=test-table [id=ec87b5e0120f4e4084adb9cee0a72369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:56.513595 28833 tablet_service.cc:1467] Processing CreateTablet for tablet a04c8e8cdb19420bb3405b252166ce26 (DEFAULT_TABLE table=test-table [id=ec87b5e0120f4e4084adb9cee0a72369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:17:56.514528 28965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a04c8e8cdb19420bb3405b252166ce26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:56.514624 28833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a04c8e8cdb19420bb3405b252166ce26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:17:56.520757 29045 tablet_bootstrap.cc:492] T a04c8e8cdb19420bb3405b252166ce26 P c231762c0bfe4a319fa0c14a1606623c: Bootstrap starting.
I20260929 03:17:56.520757 29044 tablet_bootstrap.cc:492] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4: Bootstrap starting.
I20260929 03:17:56.523319 29044 tablet_bootstrap.cc:654] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:56.523322 29045 tablet_bootstrap.cc:654] T a04c8e8cdb19420bb3405b252166ce26 P c231762c0bfe4a319fa0c14a1606623c: Neither blocks nor log segments found. Creating new log.
I20260929 03:17:56.524051 29045 log.cc:824] T a04c8e8cdb19420bb3405b252166ce26 P c231762c0bfe4a319fa0c14a1606623c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:56.524051 29044 log.cc:824] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:56.525738 29044 tablet_bootstrap.cc:492] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4: No bootstrap required, opened a new log
I20260929 03:17:56.525748 29045 tablet_bootstrap.cc:492] T a04c8e8cdb19420bb3405b252166ce26 P c231762c0bfe4a319fa0c14a1606623c: No bootstrap required, opened a new log
I20260929 03:17:56.525902 29045 ts_tablet_manager.cc:1397] T a04c8e8cdb19420bb3405b252166ce26 P c231762c0bfe4a319fa0c14a1606623c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 03:17:56.525904 29044 ts_tablet_manager.cc:1397] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 03:17:56.528990 29044 raft_consensus.cc:348] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } } peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } }
I20260929 03:17:56.528990 29045 raft_consensus.cc:348] T a04c8e8cdb19420bb3405b252166ce26 P c231762c0bfe4a319fa0c14a1606623c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } } peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } }
I20260929 03:17:56.529218 29044 raft_consensus.cc:374] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:56.529222 29045 raft_consensus.cc:374] T a04c8e8cdb19420bb3405b252166ce26 P c231762c0bfe4a319fa0c14a1606623c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:17:56.529253 29044 raft_consensus.cc:729] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92aa75a5e8e4eb2b97286955650f3f4, State: Initialized, Role: FOLLOWER
I20260929 03:17:56.529276 29045 raft_consensus.cc:729] T a04c8e8cdb19420bb3405b252166ce26 P c231762c0bfe4a319fa0c14a1606623c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c231762c0bfe4a319fa0c14a1606623c, State: Initialized, Role: FOLLOWER
I20260929 03:17:56.529721 29045 consensus_queue.cc:260] T a04c8e8cdb19420bb3405b252166ce26 P c231762c0bfe4a319fa0c14a1606623c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } } peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } }
I20260929 03:17:56.529721 29044 consensus_queue.cc:260] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } } peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } }
I20260929 03:17:56.530402 29031 heartbeater.cc:503] Master 127.13.238.190:43239 was elected leader, sending a full tablet report...
I20260929 03:17:56.530742 29045 ts_tablet_manager.cc:1428] T a04c8e8cdb19420bb3405b252166ce26 P c231762c0bfe4a319fa0c14a1606623c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 03:17:56.530742 29044 ts_tablet_manager.cc:1428] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 03:17:56.531056 28899 heartbeater.cc:503] Master 127.13.238.190:43239 was elected leader, sending a full tablet report...
I20260929 03:17:56.577165 29049 raft_consensus.cc:482] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:17:56.577360 29049 raft_consensus.cc:504] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } } peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } }
I20260929 03:17:56.578599 29049 leader_election.cc:288] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c231762c0bfe4a319fa0c14a1606623c (127.13.238.131:40529), 7ed14d80f04c482bb017fd772a1c22ae (127.13.238.129:42287)
I20260929 03:17:56.583395 28769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:56.578947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:53449 (local address 127.13.238.129:42287)
0929 03:17:56.579083 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:56.579087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:56.579352 (+ 265us) server_negotiation.cc:407] Connection header received
0929 03:17:56.579440 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:56.579443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:56.579500 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:56.579575 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:56.579818 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.580596 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.582126 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.582789 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.582884 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:56.583027 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:56.583189 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 03:17:56.583260 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:17:56.583392 29042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:56.578909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:38379 (local address 127.13.238.131:40529)
0929 03:17:56.579052 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:56.579057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:56.579072 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:17:56.579236 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:56.579239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:56.579283 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:56.579356 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:17:56.579652 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.580462 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.582105 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.582740 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.582868 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:56.582924 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:56.583190 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 03:17:56.583261 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:17:56.583426 28901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:56.578812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:40529 (local address 127.13.238.130:38379)
0929 03:17:56.579006 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:56.579021 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:56.579111 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:56.579385 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:56.579388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:56.579402 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:56.579530 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:56.579536 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.580599 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.580602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:56.581972 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:56.581984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.582885 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.582889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:56.583067 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:56.583087 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 03:17:56.583167 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 03:17:56.583273 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":144}
I20260929 03:17:56.583436 29050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:56.578858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:42287 (local address 127.13.238.130:53449)
0929 03:17:56.579263 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:56.579273 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:56.579355 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:56.579612 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:56.579615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:56.579633 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:17:56.579720 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:56.579724 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.580700 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.580702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:56.582026 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:56.582034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.582902 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.582906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:56.583055 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:56.583085 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 03:17:56.583168 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 03:17:56.583273 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":103,"thread_start_us":123,"threads_started":1}
I20260929 03:17:56.584060 28715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04c8e8cdb19420bb3405b252166ce26" candidate_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ed14d80f04c482bb017fd772a1c22ae" is_pre_election: true
I20260929 03:17:56.584072 28985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04c8e8cdb19420bb3405b252166ce26" candidate_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c231762c0bfe4a319fa0c14a1606623c" is_pre_election: true
I20260929 03:17:56.584347 28715 raft_consensus.cc:2454] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92aa75a5e8e4eb2b97286955650f3f4 in term 0.
I20260929 03:17:56.584375 28985 raft_consensus.cc:2454] T a04c8e8cdb19420bb3405b252166ce26 P c231762c0bfe4a319fa0c14a1606623c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92aa75a5e8e4eb2b97286955650f3f4 in term 0.
I20260929 03:17:56.584816 28789 leader_election.cc:302] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ed14d80f04c482bb017fd772a1c22ae, c92aa75a5e8e4eb2b97286955650f3f4; no voters:
I20260929 03:17:56.585094 29049 raft_consensus.cc:2781] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:17:56.585177 29049 raft_consensus.cc:482] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:17:56.585220 29049 raft_consensus.cc:3037] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4 [term 0 FOLLOWER]: Advancing to term 1
W20260929 03:17:56.585381 28762 tablet.cc:2369] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:17:56.586376 29049 raft_consensus.cc:504] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } } peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } }
I20260929 03:17:56.586722 29049 leader_election.cc:288] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4 [CANDIDATE]: Term 1 election: Requested vote from peers c231762c0bfe4a319fa0c14a1606623c (127.13.238.131:40529), 7ed14d80f04c482bb017fd772a1c22ae (127.13.238.129:42287)
I20260929 03:17:56.587042 28985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04c8e8cdb19420bb3405b252166ce26" candidate_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c231762c0bfe4a319fa0c14a1606623c"
I20260929 03:17:56.587112 28715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04c8e8cdb19420bb3405b252166ce26" candidate_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ed14d80f04c482bb017fd772a1c22ae"
I20260929 03:17:56.587155 28985 raft_consensus.cc:3037] T a04c8e8cdb19420bb3405b252166ce26 P c231762c0bfe4a319fa0c14a1606623c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:56.587229 28715 raft_consensus.cc:3037] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:17:56.588102 28985 raft_consensus.cc:2454] T a04c8e8cdb19420bb3405b252166ce26 P c231762c0bfe4a319fa0c14a1606623c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92aa75a5e8e4eb2b97286955650f3f4 in term 1.
I20260929 03:17:56.588225 28715 raft_consensus.cc:2454] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92aa75a5e8e4eb2b97286955650f3f4 in term 1.
I20260929 03:17:56.588394 28786 leader_election.cc:302] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c231762c0bfe4a319fa0c14a1606623c, c92aa75a5e8e4eb2b97286955650f3f4; no voters:
I20260929 03:17:56.588554 29049 raft_consensus.cc:2781] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:17:56.588760 29049 raft_consensus.cc:686] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4 [term 1 LEADER]: Becoming Leader. State: Replica: c92aa75a5e8e4eb2b97286955650f3f4, State: Running, Role: LEADER
I20260929 03:17:56.589056 29049 consensus_queue.cc:237] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } } peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } }
I20260929 03:17:56.592223 28572 catalog_manager.cc:5495] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to c92aa75a5e8e4eb2b97286955650f3f4 (127.13.238.130). New cstate: current_term: 1 leader_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } health_report { overall_health: UNKNOWN } } }
I20260929 03:17:56.614832 29040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:56.611495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56988 (local address 127.13.238.130:41871)
0929 03:17:56.611639 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:56.611643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:56.611657 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:17:56.611750 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:56.611752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:56.611806 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:56.611892 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:17:56.612102 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.612614 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.613456 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:56.613675 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:56.613761 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:56.613985 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:17:56.614470 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:17:56.614562 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:56.614641 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 03:17:56.614693 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:17:56.620481 28715 raft_consensus.cc:1261] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [term 1 FOLLOWER]: Refusing update from remote peer c92aa75a5e8e4eb2b97286955650f3f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:56.620486 28985 raft_consensus.cc:1261] T a04c8e8cdb19420bb3405b252166ce26 P c231762c0bfe4a319fa0c14a1606623c [term 1 FOLLOWER]: Refusing update from remote peer c92aa75a5e8e4eb2b97286955650f3f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:17:56.621155 29049 consensus_queue.cc:1035] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:56.621331 29051 consensus_queue.cc:1035] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:17:56.627547 29055 mvcc.cc:205] Tried to move back new op lower bound from 7334510086630682624 to 7334510086511181824. Current Snapshot: MvccSnapshot[applied={T|T < 7334510086630682624}]
I20260929 03:17:56.635365 28573 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:46616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:17:56.635533 28573 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:46616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:17:56.637683 28573 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d: Sending DeleteTablet for 3 replicas of tablet a04c8e8cdb19420bb3405b252166ce26
I20260929 03:17:56.638338 28833 tablet_service.cc:1514] Processing DeleteTablet for tablet a04c8e8cdb19420bb3405b252166ce26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:56 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:56976
I20260929 03:17:56.638454 28965 tablet_service.cc:1514] Processing DeleteTablet for tablet a04c8e8cdb19420bb3405b252166ce26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:56 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:55048
I20260929 03:17:56.638798 28695 tablet_service.cc:1514] Processing DeleteTablet for tablet a04c8e8cdb19420bb3405b252166ce26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:17:56 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:33642
I20260929 03:17:56.638854 29064 tablet_replica.cc:330] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4: stopping tablet replica
I20260929 03:17:56.639074 29066 tablet_replica.cc:330] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae: stopping tablet replica
I20260929 03:17:56.639183 29064 raft_consensus.cc:2229] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:17:56.639269 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 28611
I20260929 03:17:56.639550 29064 raft_consensus.cc:2258] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:56.641116 29064 ts_tablet_manager.cc:1905] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:56.643966 29064 ts_tablet_manager.cc:1918] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:17:56.644064 29064 log.cc:1196] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/wal/wals/a04c8e8cdb19420bb3405b252166ce26
I20260929 03:17:56.644348 29065 tablet_replica.cc:330] T a04c8e8cdb19420bb3405b252166ce26 P c231762c0bfe4a319fa0c14a1606623c: stopping tablet replica
I20260929 03:17:56.644393 29064 ts_tablet_manager.cc:1939] T a04c8e8cdb19420bb3405b252166ce26 P c92aa75a5e8e4eb2b97286955650f3f4: Deleting consensus metadata
I20260929 03:17:56.644737 29065 raft_consensus.cc:2229] T a04c8e8cdb19420bb3405b252166ce26 P c231762c0bfe4a319fa0c14a1606623c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:17:56.645124 29065 raft_consensus.cc:2258] T a04c8e8cdb19420bb3405b252166ce26 P c231762c0bfe4a319fa0c14a1606623c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:17:56.645606 28558 catalog_manager.cc:4841] TS c92aa75a5e8e4eb2b97286955650f3f4 (127.13.238.130:41871): tablet a04c8e8cdb19420bb3405b252166ce26 (table test-table [id=ec87b5e0120f4e4084adb9cee0a72369]) successfully deleted
W20260929 03:17:56.646173 28560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.238.129:42287 (error 108)
I20260929 03:17:56.646739 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 28766
I20260929 03:17:56.646870 29065 ts_tablet_manager.cc:1905] T a04c8e8cdb19420bb3405b252166ce26 P c231762c0bfe4a319fa0c14a1606623c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:17:56.648427 29041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:56.647629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:42287 (local address 127.0.0.1:33644)
0929 03:17:56.648261 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:56.648337 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.129:42287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537}
I20260929 03:17:56.649358 29065 ts_tablet_manager.cc:1918] T a04c8e8cdb19420bb3405b252166ce26 P c231762c0bfe4a319fa0c14a1606623c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:17:56.649456 29065 log.cc:1196] T a04c8e8cdb19420bb3405b252166ce26 P c231762c0bfe4a319fa0c14a1606623c: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/a04c8e8cdb19420bb3405b252166ce26
I20260929 03:17:56.649760 29065 ts_tablet_manager.cc:1939] T a04c8e8cdb19420bb3405b252166ce26 P c231762c0bfe4a319fa0c14a1606623c: Deleting consensus metadata
W20260929 03:17:56.649359 28560 catalog_manager.cc:4584] TS 7ed14d80f04c482bb017fd772a1c22ae (127.13.238.129:42287): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a04c8e8cdb19420bb3405b252166ce26: Network error: Client connection negotiation failed: client connection to 127.13.238.129:42287: connect: Connection refused (error 111)
I20260929 03:17:56.650910 28557 catalog_manager.cc:4841] TS c231762c0bfe4a319fa0c14a1606623c (127.13.238.131:40529): tablet a04c8e8cdb19420bb3405b252166ce26 (table test-table [id=ec87b5e0120f4e4084adb9cee0a72369]) successfully deleted
I20260929 03:17:56.652599 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 28902
I20260929 03:17:56.657433 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 28540
I20260929 03:17:56.662585 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:43239
--webserver_interface=127.13.238.190
--webserver_port=43197
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:43239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:56.772706 29068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:56.772957 29068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:56.772995 29068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:56.776986 29068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:17:56.777065 29068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:56.777086 29068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:56.777112 29068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:17:56.777141 29068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:17:56.782394 29068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:43239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:43239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=43197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:17:56.783393 29068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:56.784373 29068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:56.792457 29076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:56.792480 29073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:56.792480 29074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:56.792696 29068 server_base.cc:1034] running on GCE node
I20260929 03:17:56.793334 29068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:56.794428 29068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:56.795581 29068 hybrid_clock.cc:648] HybridClock initialized: now 1790651876795551 us; error 38 us; skew 500 ppm
Sep 29 03:17:56 dist-test-slave-nbjg krb5kdc[27992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651876, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:56.798241 29068 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:17:56.799088 29068 webserver.cc:460] Webserver started at http://127.13.238.190:43197/ using document root <none> and password file <none>
I20260929 03:17:56.799602 29068 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:56.799660 29068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:56.803334 29068 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:17:56.805281 29083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:56.806489 29068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:17:56.806584 29068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "678678b2d3d24b57b34c646855d8b52d"
format_stamp: "Formatted at 2026-09-29 03:17:54 on dist-test-slave-nbjg"
server_key: "801aa173609a1d676c827bde99fe34b8"
server_key_iv: "35be0706771725e6669041506764235e"
server_key_version: "encryptionkey@0"
I20260929 03:17:56.807071 29068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:56.818593 29068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:56.821170 29068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:56.821313 29068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:56.828003 29068 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:43239
I20260929 03:17:56.828248 29135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:43239 every 8 connection(s)
I20260929 03:17:56.829123 29068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:17:56.833446 29136 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:17:56.835167 29136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d: Bootstrap starting.
I20260929 03:17:56.838229 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 29068
I20260929 03:17:56.838697 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:42287
--local_ip_for_outbound_sockets=127.13.238.129
--tserver_master_addrs=127.13.238.190:43239
--webserver_port=45851
--webserver_interface=127.13.238.129
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:17:56.852793 29136 log.cc:824] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:56.864267 29136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:17:56.864607 29136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d: Bootstrap complete.
I20260929 03:17:56.868573 29136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678678b2d3d24b57b34c646855d8b52d" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43239 } }
I20260929 03:17:56.868955 29136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 678678b2d3d24b57b34c646855d8b52d, State: Initialized, Role: FOLLOWER
I20260929 03:17:56.869421 29136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678678b2d3d24b57b34c646855d8b52d" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43239 } }
I20260929 03:17:56.869535 29136 raft_consensus.cc:388] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:17:56.869598 29136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:17:56.869688 29136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [term 2 FOLLOWER]: Advancing to term 3
I20260929 03:17:56.871553 29136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678678b2d3d24b57b34c646855d8b52d" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43239 } }
I20260929 03:17:56.871855 29136 leader_election.cc:302] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 678678b2d3d24b57b34c646855d8b52d; no voters:
I20260929 03:17:56.872148 29136 leader_election.cc:288] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 03:17:56.872246 29140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [term 3 FOLLOWER]: Leader election won for term 3
I20260929 03:17:56.872431 29140 raft_consensus.cc:686] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [term 3 LEADER]: Becoming Leader. State: Replica: 678678b2d3d24b57b34c646855d8b52d, State: Running, Role: LEADER
I20260929 03:17:56.872816 29140 consensus_queue.cc:237] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678678b2d3d24b57b34c646855d8b52d" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43239 } }
I20260929 03:17:56.873399 29136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:17:56.873744 29140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 678678b2d3d24b57b34c646855d8b52d. Latest consensus state: current_term: 3 leader_uuid: "678678b2d3d24b57b34c646855d8b52d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678678b2d3d24b57b34c646855d8b52d" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43239 } } }
I20260929 03:17:56.873857 29140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:56.874230 29144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:17:56.874256 29141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "678678b2d3d24b57b34c646855d8b52d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678678b2d3d24b57b34c646855d8b52d" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43239 } } }
I20260929 03:17:56.874461 29141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 678678b2d3d24b57b34c646855d8b52d [sys.catalog]: This master's current role is: LEADER
I20260929 03:17:56.879295 29144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:17:56.880072 29144 catalog_manager.cc:1259] Loaded cluster ID: 6655e4cd99034f35a1cc590422e33d7e
I20260929 03:17:56.880133 29144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 03:17:56.881202 29144 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f03e60459ad google::LogMessage::Fail() at ??:0
@ 0x7f03e604888a google::LogMessage::SendToLog() at ??:0
@ 0x7f03e60454de google::LogMessage::Flush() at ??:0
@ 0x7f03e6046a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f03f0984fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f03f0985d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f03f0982579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f03f09c6640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f03e82973d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f03e8297c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f03e82993f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f03e8288512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f03e98cc6db start_thread at ??:0
@ 0x7f03e4b4361f clone at ??:0
W20260929 03:17:56.974972 29138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:56.975332 29138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:56.975399 29138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:56.981379 29138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:56.981472 29138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:56.981565 29138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:17:56.989326 29138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:42287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=45851
--tserver_master_addrs=127.13.238.190:43239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:17:56.990796 29138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:56.992185 29138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:57.003454 29177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:57.003456 29178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:57.003741 29138 server_base.cc:1034] running on GCE node
W20260929 03:17:57.003539 29180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:57.004310 29138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:57.004995 29138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:57.006170 29138 hybrid_clock.cc:648] HybridClock initialized: now 1790651877006147 us; error 38 us; skew 500 ppm
Sep 29 03:17:57 dist-test-slave-nbjg krb5kdc[27992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651877, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:57.008762 29138 init.cc:377] Logged in from keytab as kudu/127.13.238.129@KRBTEST.COM (short username kudu)
I20260929 03:17:57.009644 29138 webserver.cc:460] Webserver started at http://127.13.238.129:45851/ using document root <none> and password file <none>
I20260929 03:17:57.010215 29138 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:57.010301 29138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:57.014405 29138 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:17:57.016646 29187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:57.018038 29138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:17:57.018215 29138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "7ed14d80f04c482bb017fd772a1c22ae"
format_stamp: "Formatted at 2026-09-29 03:17:54 on dist-test-slave-nbjg"
server_key: "0203449a79a66624da04a0c128d8c1bf"
server_key_iv: "421357bffe42a88dedb6260c2ab2491f"
server_key_version: "encryptionkey@0"
I20260929 03:17:57.018781 29138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:57.039332 29138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:57.042524 29138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:57.042706 29138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:57.043304 29138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:57.044454 29194 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 03:17:57.049871 29138 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 03:17:57.049937 29138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260929 03:17:57.050001 29138 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 03:17:57.051754 29138 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 03:17:57.051805 29138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 03:17:57.052129 29194 tablet_bootstrap.cc:492] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae: Bootstrap starting.
I20260929 03:17:57.061407 29138 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:42287
I20260929 03:17:57.061662 29301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:42287 every 8 connection(s)
I20260929 03:17:57.062449 29138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:17:57.065634 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 29138
I20260929 03:17:57.066071 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:41871
--local_ip_for_outbound_sockets=127.13.238.130
--tserver_master_addrs=127.13.238.190:43239
--webserver_port=45469
--webserver_interface=127.13.238.130
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:17:57.074808 29194 log.cc:824] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae: Log is configured to *not* fsync() on all Append() calls
I20260929 03:17:57.081786 29194 tablet_bootstrap.cc:492] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:17:57.082180 29194 tablet_bootstrap.cc:492] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae: Bootstrap complete.
I20260929 03:17:57.083120 29194 ts_tablet_manager.cc:1397] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae: Time spent bootstrapping tablet: real 0.031s user 0.021s sys 0.008s
I20260929 03:17:57.085475 29194 raft_consensus.cc:348] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } } peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } }
I20260929 03:17:57.085889 29194 raft_consensus.cc:729] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed14d80f04c482bb017fd772a1c22ae, State: Initialized, Role: FOLLOWER
I20260929 03:17:57.086395 29194 consensus_queue.cc:260] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } } peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } }
I20260929 03:17:57.088359 29194 ts_tablet_manager.cc:1428] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260929 03:17:57.146543 29304 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:17:57.064431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.129:33821)
0929 03:17:57.064957 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:57.065011 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:57.066089 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:57.145669 (+ 79580us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":124,"threads_started":1}
W20260929 03:17:57.147148 29302 heartbeater.cc:644] Failed to heartbeat to 127.13.238.190:43239 (0 consecutive failures): Network error: Failed to ping master at 127.13.238.190:43239: Client connection negotiation failed: client connection to 127.13.238.190:43239: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 03:17:57.147945 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:57.147684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.129:53257)
0929 03:17:57.147823 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:57.147889 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 03:17:57.148679 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:57.148466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.129:55281)
0929 03:17:57.148581 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:57.148628 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 03:17:57.148841 29302 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:17:57.192057 29305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:57.192327 29305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:57.192371 29305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:57.196410 29305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:57.196472 29305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:57.196542 29305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:17:57.202047 29305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:41871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=45469
--tserver_master_addrs=127.13.238.190:43239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:17:57.203097 29305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:57.204265 29305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:57.214833 29312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:57.214857 29313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:57.214917 29305 server_base.cc:1034] running on GCE node
W20260929 03:17:57.214861 29315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:57.215416 29305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:57.216034 29305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:57.217190 29305 hybrid_clock.cc:648] HybridClock initialized: now 1790651877217170 us; error 32 us; skew 500 ppm
Sep 29 03:17:57 dist-test-slave-nbjg krb5kdc[27992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651877, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:57.219805 29305 init.cc:377] Logged in from keytab as kudu/127.13.238.130@KRBTEST.COM (short username kudu)
I20260929 03:17:57.220631 29305 webserver.cc:460] Webserver started at http://127.13.238.130:45469/ using document root <none> and password file <none>
I20260929 03:17:57.221159 29305 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:57.221225 29305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:57.225099 29305 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:17:57.227155 29322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:57.228330 29305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:17:57.228446 29305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "c92aa75a5e8e4eb2b97286955650f3f4"
format_stamp: "Formatted at 2026-09-29 03:17:55 on dist-test-slave-nbjg"
server_key: "6e0af4329db494a53b431ed7f300d266"
server_key_iv: "ed02205b4b588768e83f79a1a452a9c7"
server_key_version: "encryptionkey@0"
I20260929 03:17:57.228967 29305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:57.252506 29305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:57.255589 29305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:57.255766 29305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:57.256389 29305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:57.257292 29305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:57.257344 29305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:57.257400 29305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:57.257429 29305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:57.266053 29305 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:41871
I20260929 03:17:57.266083 29435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:41871 every 8 connection(s)
I20260929 03:17:57.267130 29305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:17:57.271091 29438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:57.269345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.130:46715)
0929 03:17:57.269795 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:57.269911 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":108,"threads_started":1}
W20260929 03:17:57.271505 29436 heartbeater.cc:644] Failed to heartbeat to 127.13.238.190:43239 (0 consecutive failures): Network error: Failed to ping master at 127.13.238.190:43239: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
I20260929 03:17:57.272066 29438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:57.271858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.130:50961)
0929 03:17:57.271963 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:57.272011 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 03:17:57.272616 29438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:57.272423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.130:51985)
0929 03:17:57.272515 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:57.272558 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 03:17:57.272764 29436 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 03:17:57.274967 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 29305
I20260929 03:17:57.275323 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:40529
--local_ip_for_outbound_sockets=127.13.238.131
--tserver_master_addrs=127.13.238.190:43239
--webserver_port=39617
--webserver_interface=127.13.238.131
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:17:57.383267 29439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:17:57.383517 29439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:17:57.383558 29439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:17:57.387409 29439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:17:57.387455 29439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:17:57.387509 29439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:17:57.392500 29439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:40529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=39617
--tserver_master_addrs=127.13.238.190:43239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:17:57.393446 29439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:17:57.394425 29439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:17:57.403630 29445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:57.403661 29439 server_base.cc:1034] running on GCE node
W20260929 03:17:57.403605 29447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:17:57.403590 29444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:17:57.404232 29439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:17:57.404840 29439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:17:57.405997 29439 hybrid_clock.cc:648] HybridClock initialized: now 1790651877405976 us; error 30 us; skew 500 ppm
Sep 29 03:17:57 dist-test-slave-nbjg krb5kdc[27992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651877, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:17:57.408525 29439 init.cc:377] Logged in from keytab as kudu/127.13.238.131@KRBTEST.COM (short username kudu)
I20260929 03:17:57.409343 29439 webserver.cc:460] Webserver started at http://127.13.238.131:39617/ using document root <none> and password file <none>
I20260929 03:17:57.409852 29439 fs_manager.cc:359] Metadata directory not provided
I20260929 03:17:57.409907 29439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:17:57.413683 29439 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:17:57.415727 29454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:57.416989 29439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 03:17:57.417093 29439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "c231762c0bfe4a319fa0c14a1606623c"
format_stamp: "Formatted at 2026-09-29 03:17:55 on dist-test-slave-nbjg"
server_key: "a3c0f0e805a2d00fa717ad17433d09aa"
server_key_iv: "b004142a762f46847541a3249a903237"
server_key_version: "encryptionkey@0"
I20260929 03:17:57.417632 29439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:17:57.439291 29439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:17:57.442076 29439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:17:57.442276 29439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:17:57.442857 29439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:17:57.443648 29439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:17:57.443692 29439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:57.443743 29439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:17:57.443771 29439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:17:57.452054 29439 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:40529
I20260929 03:17:57.452085 29567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:40529 every 8 connection(s)
I20260929 03:17:57.453023 29439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:17:57.456826 29570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:57.455083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.131:53649)
0929 03:17:57.455542 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:57.455663 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":115,"threads_started":1}
W20260929 03:17:57.457301 29568 heartbeater.cc:644] Failed to heartbeat to 127.13.238.190:43239 (0 consecutive failures): Network error: Failed to ping master at 127.13.238.190:43239: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
I20260929 03:17:57.457864 29570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:57.457664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.131:41041)
0929 03:17:57.457770 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:57.457814 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 03:17:57.458451 29570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:57.458271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.131:53733)
0929 03:17:57.458359 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:57.458402 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 03:17:57.458596 29568 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 03:17:57.461130 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 29439
I20260929 03:17:58.149947 29572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:58.149478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.129:47957)
0929 03:17:58.149764 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:58.149829 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":84,"threads_started":1}
I20260929 03:17:58.273665 29573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:58.273267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.130:60077)
0929 03:17:58.273512 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:58.273566 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":80,"threads_started":1}
I20260929 03:17:58.459509 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:58.459089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.131:42485)
0929 03:17:58.459361 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:58.459416 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":108,"threads_started":1}
I20260929 03:17:58.724834 29575 raft_consensus.cc:482] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:17:58.725067 29575 raft_consensus.cc:504] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } } peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } }
I20260929 03:17:58.726732 29575 leader_election.cc:288] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c92aa75a5e8e4eb2b97286955650f3f4 (127.13.238.130:41871), c231762c0bfe4a319fa0c14a1606623c (127.13.238.131:40529)
Sep 29 03:17:58 dist-test-slave-nbjg krb5kdc[27992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651877, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for kudu/127.13.238.131@KRBTEST.COM
Sep 29 03:17:58 dist-test-slave-nbjg krb5kdc[27992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651877, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for kudu/127.13.238.130@KRBTEST.COM
I20260929 03:17:58.743649 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:58.726513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:41871 (local address 127.13.238.129:43907)
0929 03:17:58.726800 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:58.726814 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:58.726913 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:58.729049 (+ 2136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:58.729056 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:58.729631 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:58.730482 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:58.730506 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:58.732417 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:58.732420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:58.733912 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:58.733921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:58.735160 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:58.735164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:58.735462 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:58.736149 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:58.736176 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:58.738373 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:58.740700 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:58.740704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:58.740706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:58.741114 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:58.741482 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:58.741485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:58.741487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:58.741722 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:58.742516 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:58.742519 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:58.742852 (+ 333us) client_negotiation.cc:769] Sending connection context
0929 03:17:58.743082 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 03:17:58.743472 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":507,"spinlock_wait_cycles":23424,"thread_start_us":67,"threads_started":1}
I20260929 03:17:58.743721 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:58.726707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:43907 (local address 127.13.238.130:41871)
0929 03:17:58.727282 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:58.727301 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:58.727331 (+ 30us) server_negotiation.cc:407] Connection header received
0929 03:17:58.728103 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:58.728112 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:58.728580 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:58.728920 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:58.730681 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:58.732266 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:58.734025 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:58.734816 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:58.734980 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:58.738575 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:58.738583 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:58.738601 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:58.738633 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:58.740580 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:58.741229 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:58.741236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:58.741242 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:58.741388 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:58.741827 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:58.741830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:58.741832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:58.742392 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:58.742523 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:58.743234 (+ 711us) server_negotiation.cc:299] Negotiation successful
0929 03:17:58.743534 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":82,"threads_started":1}
I20260929 03:17:58.743718 29579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:58.726820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:56617 (local address 127.13.238.131:40529)
0929 03:17:58.727141 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0929 03:17:58.727166 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 03:17:58.727213 (+ 47us) server_negotiation.cc:407] Connection header received
0929 03:17:58.728103 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:17:58.728112 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:17:58.728580 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:17:58.728920 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:17:58.730681 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:58.732266 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:58.734025 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:17:58.734816 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:17:58.734980 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:58.738236 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:17:58.738244 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:17:58.738261 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:17:58.738289 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:17:58.740393 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:58.741229 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:58.741236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:58.741242 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:58.741388 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:17:58.741827 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:17:58.741830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:17:58.741832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:17:58.742392 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:17:58.742523 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 03:17:58.743234 (+ 711us) server_negotiation.cc:299] Negotiation successful
0929 03:17:58.743534 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":114,"threads_started":1}
I20260929 03:17:58.743649 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:58.726538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:40529 (local address 127.13.238.129:56617)
0929 03:17:58.727067 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:58.727077 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:17:58.727151 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:17:58.729049 (+ 1898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:17:58.729056 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:17:58.729631 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:17:58.730482 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:58.730508 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:58.732417 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:58.732420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:58.733910 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:17:58.733921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:17:58.735160 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:17:58.735164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:17:58.735462 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:17:58.736149 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:17:58.736189 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:17:58.737945 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:17:58.740516 (+ 2571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:58.740522 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:58.740534 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:58.741114 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:58.741482 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:17:58.741485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:17:58.741487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:17:58.741722 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:17:58.742500 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:17:58.742507 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:17:58.742852 (+ 345us) client_negotiation.cc:769] Sending connection context
0929 03:17:58.743099 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 03:17:58.743407 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":256,"spinlock_wait_cycles":89728,"thread_start_us":63,"threads_started":1}
W20260929 03:17:58.746338 29188 leader_election.cc:341] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c231762c0bfe4a319fa0c14a1606623c (127.13.238.131:40529): Not found: Tablet not found: a04c8e8cdb19420bb3405b252166ce26
W20260929 03:17:58.746472 29189 leader_election.cc:341] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c92aa75a5e8e4eb2b97286955650f3f4 (127.13.238.130:41871): Not found: Tablet not found: a04c8e8cdb19420bb3405b252166ce26
I20260929 03:17:58.746568 29189 leader_election.cc:302] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ed14d80f04c482bb017fd772a1c22ae; no voters: c231762c0bfe4a319fa0c14a1606623c, c92aa75a5e8e4eb2b97286955650f3f4
I20260929 03:17:58.746836 29575 raft_consensus.cc:2726] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:17:59.151019 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.150715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.129:52361)
0929 03:17:59.150870 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.150931 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 03:17:59.274752 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.274347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.130:51451)
0929 03:17:59.274600 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.274666 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":69,"threads_started":1}
I20260929 03:17:59.460703 29581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:17:59.460275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.131:57447)
0929 03:17:59.460542 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 03:17:59.460602 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":71,"threads_started":1}
I20260929 03:18:00.152206 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:00.151772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.129:51915)
0929 03:18:00.152039 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:00.152104 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":86,"threads_started":1}
I20260929 03:18:00.255825 29583 raft_consensus.cc:482] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:18:00.255959 29583 raft_consensus.cc:504] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } } peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } }
I20260929 03:18:00.256421 29583 leader_election.cc:288] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c92aa75a5e8e4eb2b97286955650f3f4 (127.13.238.130:41871), c231762c0bfe4a319fa0c14a1606623c (127.13.238.131:40529)
W20260929 03:18:00.257272 29188 leader_election.cc:341] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c231762c0bfe4a319fa0c14a1606623c (127.13.238.131:40529): Not found: Tablet not found: a04c8e8cdb19420bb3405b252166ce26
W20260929 03:18:00.257385 29189 leader_election.cc:341] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c92aa75a5e8e4eb2b97286955650f3f4 (127.13.238.130:41871): Not found: Tablet not found: a04c8e8cdb19420bb3405b252166ce26
I20260929 03:18:00.257469 29189 leader_election.cc:302] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ed14d80f04c482bb017fd772a1c22ae; no voters: c231762c0bfe4a319fa0c14a1606623c, c92aa75a5e8e4eb2b97286955650f3f4
I20260929 03:18:00.257628 29583 raft_consensus.cc:2726] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:18:00.276043 29584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:00.275498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.130:59283)
0929 03:18:00.275810 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:00.275869 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":85,"threads_started":1}
I20260929 03:18:00.461825 29585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:00.461375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.131:32881)
0929 03:18:00.461657 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:00.461723 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":104,"threads_started":1}
I20260929 03:18:01.153445 29586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:01.153029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.129:54597)
0929 03:18:01.153288 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:01.153347 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":99,"threads_started":1}
I20260929 03:18:01.277201 29587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:01.276761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.130:50297)
0929 03:18:01.277029 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:01.277111 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":87,"threads_started":1}
I20260929 03:18:01.462952 29588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:01.462555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.131:34063)
0929 03:18:01.462809 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:01.462862 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":86,"threads_started":1}
I20260929 03:18:02.154914 29589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:02.154437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.129:60473)
0929 03:18:02.154745 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:02.154811 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":94,"threads_started":1}
I20260929 03:18:02.278398 29590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:02.277953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.130:45623)
0929 03:18:02.278243 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:02.278302 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
I20260929 03:18:02.464063 29591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:02.463645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.131:35129)
0929 03:18:02.463906 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:02.463964 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":71,"threads_started":1}
I20260929 03:18:02.696120 29592 raft_consensus.cc:482] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:18:02.696244 29592 raft_consensus.cc:504] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } } peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } }
I20260929 03:18:02.696720 29592 leader_election.cc:288] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c92aa75a5e8e4eb2b97286955650f3f4 (127.13.238.130:41871), c231762c0bfe4a319fa0c14a1606623c (127.13.238.131:40529)
W20260929 03:18:02.697654 29189 leader_election.cc:341] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c92aa75a5e8e4eb2b97286955650f3f4 (127.13.238.130:41871): Not found: Tablet not found: a04c8e8cdb19420bb3405b252166ce26
W20260929 03:18:02.697775 29188 leader_election.cc:341] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c231762c0bfe4a319fa0c14a1606623c (127.13.238.131:40529): Not found: Tablet not found: a04c8e8cdb19420bb3405b252166ce26
I20260929 03:18:02.697952 29188 leader_election.cc:302] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ed14d80f04c482bb017fd772a1c22ae; no voters: c231762c0bfe4a319fa0c14a1606623c, c92aa75a5e8e4eb2b97286955650f3f4
I20260929 03:18:02.698166 29592 raft_consensus.cc:2726] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:18:03.156064 29593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.155633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.129:49173)
0929 03:18:03.155896 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.155955 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":80,"threads_started":1}
I20260929 03:18:03.279516 29594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.279082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.130:40851)
0929 03:18:03.279357 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.279422 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":82,"threads_started":1}
I20260929 03:18:03.465178 29595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:03.464790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.131:51115)
0929 03:18:03.465031 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:03.465085 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
I20260929 03:18:04.157390 29596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:04.156858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.129:60279)
0929 03:18:04.157227 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:04.157291 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":145,"threads_started":1}
I20260929 03:18:04.280851 29597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:04.280327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.130:56913)
0929 03:18:04.280665 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:04.280753 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":79,"threads_started":1}
I20260929 03:18:04.466538 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:04.465937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.131:52161)
0929 03:18:04.466324 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:04.466398 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":120,"threads_started":1}
I20260929 03:18:05.158653 29599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.158223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.129:50287)
0929 03:18:05.158485 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:05.158553 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":67,"threads_started":1}
I20260929 03:18:05.282054 29600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.281635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.130:49861)
0929 03:18:05.281894 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:05.281955 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":79,"threads_started":1}
I20260929 03:18:05.467669 29601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:05.467236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.131:52873)
0929 03:18:05.467513 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:05.467571 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":94,"threads_started":1}
I20260929 03:18:05.940169 29602 raft_consensus.cc:482] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:18:05.940320 29602 raft_consensus.cc:504] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } } peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } }
I20260929 03:18:05.940766 29602 leader_election.cc:288] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c92aa75a5e8e4eb2b97286955650f3f4 (127.13.238.130:41871), c231762c0bfe4a319fa0c14a1606623c (127.13.238.131:40529)
W20260929 03:18:05.941524 29189 leader_election.cc:341] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c92aa75a5e8e4eb2b97286955650f3f4 (127.13.238.130:41871): Not found: Tablet not found: a04c8e8cdb19420bb3405b252166ce26
W20260929 03:18:05.941630 29188 leader_election.cc:341] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c231762c0bfe4a319fa0c14a1606623c (127.13.238.131:40529): Not found: Tablet not found: a04c8e8cdb19420bb3405b252166ce26
I20260929 03:18:05.941691 29188 leader_election.cc:302] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ed14d80f04c482bb017fd772a1c22ae; no voters: c231762c0bfe4a319fa0c14a1606623c, c92aa75a5e8e4eb2b97286955650f3f4
I20260929 03:18:05.941854 29602 raft_consensus.cc:2726] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:18:06.159899 29603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:06.159414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.129:60139)
0929 03:18:06.159737 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:06.159795 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":116,"threads_started":1}
I20260929 03:18:06.283222 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:06.282783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.130:46517)
0929 03:18:06.283044 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:06.283116 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":75,"threads_started":1}
I20260929 03:18:06.468818 29605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:06.468362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.131:58863)
0929 03:18:06.468625 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:06.468683 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":76,"threads_started":1}
I20260929 03:18:07.161157 29606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:07.160759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.129:52789)
0929 03:18:07.160992 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:07.161055 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":72,"threads_started":1}
I20260929 03:18:07.284431 29607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:07.283951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.130:42441)
0929 03:18:07.284261 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:07.284324 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":85,"threads_started":1}
I20260929 03:18:07.469986 29608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:07.469585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.131:42013)
0929 03:18:07.469831 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:07.469893 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":84,"threads_started":1}
I20260929 03:18:08.109946 29609 raft_consensus.cc:482] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:18:08.110083 29609 raft_consensus.cc:504] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92aa75a5e8e4eb2b97286955650f3f4" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 41871 } } peers { permanent_uuid: "c231762c0bfe4a319fa0c14a1606623c" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40529 } } peers { permanent_uuid: "7ed14d80f04c482bb017fd772a1c22ae" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42287 } }
I20260929 03:18:08.110627 29609 leader_election.cc:288] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c92aa75a5e8e4eb2b97286955650f3f4 (127.13.238.130:41871), c231762c0bfe4a319fa0c14a1606623c (127.13.238.131:40529)
W20260929 03:18:08.111265 29188 leader_election.cc:341] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c231762c0bfe4a319fa0c14a1606623c (127.13.238.131:40529): Not found: Tablet not found: a04c8e8cdb19420bb3405b252166ce26
W20260929 03:18:08.111358 29189 leader_election.cc:341] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c92aa75a5e8e4eb2b97286955650f3f4 (127.13.238.130:41871): Not found: Tablet not found: a04c8e8cdb19420bb3405b252166ce26
I20260929 03:18:08.111451 29189 leader_election.cc:302] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ed14d80f04c482bb017fd772a1c22ae; no voters: c231762c0bfe4a319fa0c14a1606623c, c92aa75a5e8e4eb2b97286955650f3f4
I20260929 03:18:08.111589 29609 raft_consensus.cc:2726] T a04c8e8cdb19420bb3405b252166ce26 P 7ed14d80f04c482bb017fd772a1c22ae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:18:08.162421 29610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:08.161910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.129:38823)
0929 03:18:08.162250 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:08.162319 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":78,"threads_started":1}
I20260929 03:18:08.285561 29611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:08.285138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.130:38643)
0929 03:18:08.285392 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:08.285467 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":69,"threads_started":1}
I20260929 03:18:08.471063 29612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:08.470698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.131:54097)
0929 03:18:08.470916 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:08.470967 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":67,"threads_started":1}
I20260929 03:18:09.163780 29613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:09.163294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.129:49855)
0929 03:18:09.163589 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:09.163660 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":84,"threads_started":1}
I20260929 03:18:09.286749 29614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:09.286330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.130:60965)
0929 03:18:09.286591 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:09.286651 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
I20260929 03:18:09.472164 29615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:09.471749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.131:53373)
0929 03:18:09.472013 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:09.472070 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
I20260929 03:18:10.165047 29616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:10.164585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.129:48005)
0929 03:18:10.164843 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:10.164914 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260929 03:18:10.287902 29617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:10.287454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.130:42317)
0929 03:18:10.287733 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:10.287793 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":76,"threads_started":1}
I20260929 03:18:10.473317 29618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:10.472887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.131:60657)
0929 03:18:10.473165 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:10.473223 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":89,"threads_started":1}
I20260929 03:18:11.166319 29619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:11.165853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.129:40231)
0929 03:18:11.166131 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:11.166188 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":80,"threads_started":1}
I20260929 03:18:11.289016 29620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:11.288630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.130:40725)
0929 03:18:11.288862 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:11.288919 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":75,"threads_started":1}
I20260929 03:18:11.474557 29621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:11.474068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.131:47785)
0929 03:18:11.474390 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:11.474454 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":115,"threads_started":1}
I20260929 03:18:12.167613 29622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:12.167218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.129:44417)
0929 03:18:12.167453 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:12.167510 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":70,"threads_started":1}
I20260929 03:18:12.290313 29623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:12.289853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.130:56403)
0929 03:18:12.290116 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:12.290178 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":71,"threads_started":1}
I20260929 03:18:12.467916 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 29138
I20260929 03:18:12.473505 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 29305
I20260929 03:18:12.475819 29624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:12.475362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43239 (local address 127.13.238.131:43161)
0929 03:18:12.475643 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:12.475710 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":59,"threads_started":1}
I20260929 03:18:12.478668 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 29439
2026-09-29T03:18:12Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19041 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:18:12 dist-test-slave-nbjg krb5kdc[29627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:18:12 dist-test-slave-nbjg krb5kdc[29627](info): set up 2 sockets
Sep 29 03:18:12 dist-test-slave-nbjg krb5kdc[29627](info): commencing operation
krb5kdc: starting...
W20260929 03:18:13.438354 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:18:13 dist-test-slave-nbjg krb5kdc[29627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:18:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:18:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:18:13.552644 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:42831
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:42831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:18:13.662182 29640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:13.662472 29640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:13.662513 29640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:13.666440 29640 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 03:18:13.666489 29640 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 03:18:13.666504 29640 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 03:18:13.666517 29640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:18:13.666533 29640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:13.666549 29640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:13.666563 29640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:18:13.666580 29640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:18:13.671801 29640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:42831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:42831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:18:13.672912 29640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:13.673951 29640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:13.682349 29646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:13.682376 29648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:13.682376 29645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:13.682591 29640 server_base.cc:1034] running on GCE node
I20260929 03:18:13.683177 29640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:13.684260 29640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:13.685420 29640 hybrid_clock.cc:648] HybridClock initialized: now 1790651893685404 us; error 30 us; skew 500 ppm
Sep 29 03:18:13 dist-test-slave-nbjg krb5kdc[29627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651893, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:13.688038 29640 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:18:13.688867 29640 webserver.cc:460] Webserver started at http://127.13.238.190:33565/ using document root <none> and password file <none>
I20260929 03:18:13.689394 29640 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:13.689458 29640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:13.689651 29640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:18:13.691702 29640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "6a674960548b41ef9bdd6508f2e421ce"
format_stamp: "Formatted at 2026-09-29 03:18:13 on dist-test-slave-nbjg"
server_key: "491f9449b5027cd553d1083c533c5607"
server_key_iv: "01ca876096e726f385b02b4b205811ce"
server_key_version: "encryptionkey@0"
I20260929 03:18:13.692224 29640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "6a674960548b41ef9bdd6508f2e421ce"
format_stamp: "Formatted at 2026-09-29 03:18:13 on dist-test-slave-nbjg"
server_key: "491f9449b5027cd553d1083c533c5607"
server_key_iv: "01ca876096e726f385b02b4b205811ce"
server_key_version: "encryptionkey@0"
I20260929 03:18:13.695916 29640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260929 03:18:13.698293 29655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:13.699862 29640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:18:13.699960 29640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "6a674960548b41ef9bdd6508f2e421ce"
format_stamp: "Formatted at 2026-09-29 03:18:13 on dist-test-slave-nbjg"
server_key: "491f9449b5027cd553d1083c533c5607"
server_key_iv: "01ca876096e726f385b02b4b205811ce"
server_key_version: "encryptionkey@0"
I20260929 03:18:13.700052 29640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:13.731739 29640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:13.734709 29640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:13.734863 29640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:13.741739 29640 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:42831
I20260929 03:18:13.742010 29707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:42831 every 8 connection(s)
I20260929 03:18:13.742772 29640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:18:13.745623 29708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:13.748471 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 29640
I20260929 03:18:13.748584 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:18:13.748832 14266 external_mini_cluster.cc:1428] Setting key 6335be639f2856ff79fb221679167c2d
I20260929 03:18:13.751575 29708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce: Bootstrap starting.
I20260929 03:18:13.754113 29708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:13.755031 29708 log.cc:824] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce: Log is configured to *not* fsync() on all Append() calls
I20260929 03:18:13.757367 29708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce: No bootstrap required, opened a new log
Sep 29 03:18:13 dist-test-slave-nbjg krb5kdc[29627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:18:13.761196 29708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a674960548b41ef9bdd6508f2e421ce" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42831 } }
I20260929 03:18:13.761471 29708 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:13.761562 29708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a674960548b41ef9bdd6508f2e421ce, State: Initialized, Role: FOLLOWER
I20260929 03:18:13.762039 29708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a674960548b41ef9bdd6508f2e421ce" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42831 } }
I20260929 03:18:13.762174 29708 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:18:13.762264 29708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:18:13.762362 29708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:13.763288 29708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a674960548b41ef9bdd6508f2e421ce" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42831 } }
I20260929 03:18:13.763631 29708 leader_election.cc:302] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a674960548b41ef9bdd6508f2e421ce; no voters:
I20260929 03:18:13.763839 29711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:13.750163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41390 (local address 127.13.238.190:42831)
0929 03:18:13.750609 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:13.750626 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:13.750662 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:18:13.751303 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:13.751321 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:13.751669 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:13.751966 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:13.752820 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:13.753788 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:13.754979 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:13.755587 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:13.755747 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:13.758393 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:13.758399 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:13.758418 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:13.758449 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:13.760877 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:13.761295 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:13.761300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:13.761306 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:13.761447 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:13.761738 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:13.761742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:13.761744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:13.762180 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:13.762504 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:13.762732 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 03:18:13.763013 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":110,"threads_started":1}
I20260929 03:18:13.763916 29708 leader_election.cc:288] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:18:13.764029 29713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:18:13.764216 29713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [term 1 LEADER]: Becoming Leader. State: Replica: 6a674960548b41ef9bdd6508f2e421ce, State: Running, Role: LEADER
I20260929 03:18:13.764532 29713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a674960548b41ef9bdd6508f2e421ce" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42831 } }
I20260929 03:18:13.765197 29708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:18:13.766248 29714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a674960548b41ef9bdd6508f2e421ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a674960548b41ef9bdd6508f2e421ce" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42831 } } }
I20260929 03:18:13.766376 29714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [sys.catalog]: This master's current role is: LEADER
I20260929 03:18:13.766717 29715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a674960548b41ef9bdd6508f2e421ce. Latest consensus state: current_term: 1 leader_uuid: "6a674960548b41ef9bdd6508f2e421ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a674960548b41ef9bdd6508f2e421ce" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42831 } } }
I20260929 03:18:13.766844 29715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [sys.catalog]: This master's current role is: LEADER
I20260929 03:18:13.766772 29722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:18:13.769542 29722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:18:13.775431 29722 catalog_manager.cc:1347] Generated new cluster ID: 3dd2aa39bee746e5aea28c03060e1e39
I20260929 03:18:13.775490 29722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:18:13.782099 29722 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:18:13.783008 29722 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:18:13.790791 29722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce: Generated new TSK 0
I20260929 03:18:13.791544 29722 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
I20260929 03:18:13.825693 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:0
--local_ip_for_outbound_sockets=127.13.238.129
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:42831
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:18:13.934406 29734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:13.934667 29734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:13.934713 29734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:13.938747 29734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:13.938804 29734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:13.938867 29734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:18:13.944062 29734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:18:13.945081 29734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:13.946097 29734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:13.955291 29742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:13.955281 29740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:13.955300 29739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:13.955673 29734 server_base.cc:1034] running on GCE node
I20260929 03:18:13.956049 29734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:13.956655 29734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:13.957839 29734 hybrid_clock.cc:648] HybridClock initialized: now 1790651893957816 us; error 32 us; skew 500 ppm
Sep 29 03:18:13 dist-test-slave-nbjg krb5kdc[29627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651893, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:13.960448 29734 init.cc:377] Logged in from keytab as kudu/127.13.238.129@KRBTEST.COM (short username kudu)
I20260929 03:18:13.961278 29734 webserver.cc:460] Webserver started at http://127.13.238.129:45617/ using document root <none> and password file <none>
I20260929 03:18:13.961812 29734 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:13.961874 29734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:13.962054 29734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:18:13.964094 29734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/data/instance:
uuid: "eb3a7ddf42d54d05a6638a441d6c48e0"
format_stamp: "Formatted at 2026-09-29 03:18:13 on dist-test-slave-nbjg"
server_key: "447755d56fbd7aa9d1da9953ed2addfd"
server_key_iv: "eef7261c1439168672bdbbb926395303"
server_key_version: "encryptionkey@0"
I20260929 03:18:13.964628 29734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance:
uuid: "eb3a7ddf42d54d05a6638a441d6c48e0"
format_stamp: "Formatted at 2026-09-29 03:18:13 on dist-test-slave-nbjg"
server_key: "447755d56fbd7aa9d1da9953ed2addfd"
server_key_iv: "eef7261c1439168672bdbbb926395303"
server_key_version: "encryptionkey@0"
I20260929 03:18:13.968398 29734 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 03:18:13.970741 29749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:13.971931 29734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:18:13.972035 29734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "eb3a7ddf42d54d05a6638a441d6c48e0"
format_stamp: "Formatted at 2026-09-29 03:18:13 on dist-test-slave-nbjg"
server_key: "447755d56fbd7aa9d1da9953ed2addfd"
server_key_iv: "eef7261c1439168672bdbbb926395303"
server_key_version: "encryptionkey@0"
I20260929 03:18:13.972129 29734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:13.989173 29734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:13.992008 29734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:13.992187 29734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:13.992772 29734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:18:13.993587 29734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:18:13.993640 29734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:13.993703 29734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:18:13.993727 29734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:14.002063 29734 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:43999
I20260929 03:18:14.002126 29862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:43999 every 8 connection(s)
I20260929 03:18:14.003140 29734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:18:14.011757 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 29734
I20260929 03:18:14.011888 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance
I20260929 03:18:14.012168 14266 external_mini_cluster.cc:1428] Setting key 6e5d7fff45975083fbf0b379c700f7d7
Sep 29 03:18:14 dist-test-slave-nbjg krb5kdc[29627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651893, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:18:14.016476 29711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:14.004964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:38901 (local address 127.13.238.190:42831)
0929 03:18:14.005106 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:14.005110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:14.006146 (+ 1036us) server_negotiation.cc:407] Connection header received
0929 03:18:14.007080 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:14.007084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:14.007144 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:14.007217 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:14.008414 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.008939 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.010056 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.010339 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.010425 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:14.012915 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:14.012920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:14.012923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:14.012948 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:14.014530 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:14.015021 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:14.015025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:14.015026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:14.015104 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:14.015416 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:14.015420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:14.015422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:14.015602 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:14.015687 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:14.016173 (+ 486us) server_negotiation.cc:299] Negotiation successful
0929 03:18:14.016307 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:18:14.017331 29865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:14.005337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.129:38901)
0929 03:18:14.005986 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:14.006036 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:14.006867 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:14.007404 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:14.007411 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:14.007784 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:14.008266 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:14.008279 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.009056 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.009060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:14.009929 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:14.009938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.010491 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.010496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:14.010661 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:14.011218 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:14.011234 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:14.012746 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:14.014644 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:14.014650 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:14.014667 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:14.014921 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:14.015184 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:14.015187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:14.015189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:14.015332 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:14.015699 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:14.015706 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:14.015950 (+ 244us) client_negotiation.cc:769] Sending connection context
0929 03:18:14.016118 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 03:18:14.016439 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":114,"threads_started":1}
I20260929 03:18:14.018586 29863 heartbeater.cc:347] Connected to a master server at 127.13.238.190:42831
I20260929 03:18:14.018864 29863 heartbeater.cc:464] Registering TS with master...
I20260929 03:18:14.019488 29863 heartbeater.cc:511] Master 127.13.238.190:42831 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.238.130@KRBTEST.COM; defaulting to no policy
I20260929 03:18:14.021143 29672 ts_manager.cc:194] Registered new tserver with Master: eb3a7ddf42d54d05a6638a441d6c48e0 (127.13.238.129:43999)
I20260929 03:18:14.022393 29672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.129@KRBTEST.COM'} at 127.13.238.129:38901
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
I20260929 03:18:14.039426 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:0
--local_ip_for_outbound_sockets=127.13.238.130
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:42831
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:18:14.148218 29868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:14.148496 29868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:14.148556 29868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:14.152554 29868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:14.152614 29868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:14.152688 29868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:18:14.157932 29868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:18:14.159049 29868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:14.160095 29868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:14.169792 29876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:14.169791 29873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:14.169792 29874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:14.170145 29868 server_base.cc:1034] running on GCE node
I20260929 03:18:14.170559 29868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:14.171191 29868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:14.172382 29868 hybrid_clock.cc:648] HybridClock initialized: now 1790651894172342 us; error 45 us; skew 500 ppm
Sep 29 03:18:14 dist-test-slave-nbjg krb5kdc[29627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651894, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:14.175097 29868 init.cc:377] Logged in from keytab as kudu/127.13.238.130@KRBTEST.COM (short username kudu)
I20260929 03:18:14.175916 29868 webserver.cc:460] Webserver started at http://127.13.238.130:37135/ using document root <none> and password file <none>
I20260929 03:18:14.176435 29868 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:14.176491 29868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:14.176661 29868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:18:14.178723 29868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/data/instance:
uuid: "2a75ce578b5e4782980fdb5857faaf58"
format_stamp: "Formatted at 2026-09-29 03:18:14 on dist-test-slave-nbjg"
server_key: "c3fab53dc74dfa61b85710d05a7355e2"
server_key_iv: "77a035d42472b51565293e2b0fde7b1c"
server_key_version: "encryptionkey@0"
I20260929 03:18:14.179280 29868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance:
uuid: "2a75ce578b5e4782980fdb5857faaf58"
format_stamp: "Formatted at 2026-09-29 03:18:14 on dist-test-slave-nbjg"
server_key: "c3fab53dc74dfa61b85710d05a7355e2"
server_key_iv: "77a035d42472b51565293e2b0fde7b1c"
server_key_version: "encryptionkey@0"
I20260929 03:18:14.182915 29868 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:18:14.185391 29883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:14.186767 29868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:18:14.186870 29868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "2a75ce578b5e4782980fdb5857faaf58"
format_stamp: "Formatted at 2026-09-29 03:18:14 on dist-test-slave-nbjg"
server_key: "c3fab53dc74dfa61b85710d05a7355e2"
server_key_iv: "77a035d42472b51565293e2b0fde7b1c"
server_key_version: "encryptionkey@0"
I20260929 03:18:14.186980 29868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:14.231105 29868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:14.234414 29868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:14.234576 29868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:14.235133 29868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:18:14.236004 29868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:18:14.236060 29868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:14.236101 29868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:18:14.236121 29868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:14.244688 29868 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:42483
I20260929 03:18:14.244719 29996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:42483 every 8 connection(s)
I20260929 03:18:14.245589 29868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:18:14 dist-test-slave-nbjg krb5kdc[29627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651894, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:18:14.255831 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 29868
I20260929 03:18:14.256023 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance
I20260929 03:18:14.256353 14266 external_mini_cluster.cc:1428] Setting key e9d09f17ed67d04b927d3afa70597fc8
I20260929 03:18:14.258980 29711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:14.247354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:40007 (local address 127.13.238.190:42831)
0929 03:18:14.247504 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:14.247508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:14.248461 (+ 953us) server_negotiation.cc:407] Connection header received
0929 03:18:14.249341 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:14.249345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:14.249400 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:14.249478 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:14.250758 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.251286 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.252384 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.252605 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.252686 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:14.255200 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:14.255204 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:14.255206 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:14.255232 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:14.256771 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:14.257337 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:14.257340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:14.257342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:14.257441 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:14.257816 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:14.257819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:14.257821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:14.258033 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:14.258138 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:14.258702 (+ 564us) server_negotiation.cc:299] Negotiation successful
0929 03:18:14.258831 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 03:18:14.259788 29999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:14.247677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.130:40007)
0929 03:18:14.248308 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:14.248350 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:14.249108 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:14.249654 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:14.249661 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:14.250065 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:14.250595 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:14.250609 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.251396 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.251399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:14.252274 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:14.252283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.252765 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.252768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:14.252958 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:14.253511 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:14.253528 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:14.255021 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:14.256909 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:14.256916 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:14.256933 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:14.257244 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:14.257527 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:14.257530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:14.257532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:14.257707 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:14.258160 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:14.258166 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:14.258459 (+ 293us) client_negotiation.cc:769] Sending connection context
0929 03:18:14.258626 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 03:18:14.258884 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":106,"threads_started":1}
I20260929 03:18:14.260870 29997 heartbeater.cc:347] Connected to a master server at 127.13.238.190:42831
I20260929 03:18:14.261108 29997 heartbeater.cc:464] Registering TS with master...
I20260929 03:18:14.261647 29997 heartbeater.cc:511] Master 127.13.238.190:42831 requested a full tablet report, sending...
I20260929 03:18:14.262838 29672 ts_manager.cc:194] Registered new tserver with Master: 2a75ce578b5e4782980fdb5857faaf58 (127.13.238.130:42483)
I20260929 03:18:14.263438 29672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.130@KRBTEST.COM'} at 127.13.238.130:40007
WARNING: no policy specified for kudu/127.13.238.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
I20260929 03:18:14.284833 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:0
--local_ip_for_outbound_sockets=127.13.238.131
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:42831
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:18:14.394052 30002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:14.394328 30002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:14.394388 30002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:14.398378 30002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:14.398442 30002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:14.398522 30002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:18:14.403834 30002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:18:14.404923 30002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:14.405926 30002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:18:14.415230 30002 server_base.cc:1034] running on GCE node
W20260929 03:18:14.415191 30010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:14.415195 30007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:14.415199 30008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:14.415839 30002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:14.416455 30002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:14.417661 30002 hybrid_clock.cc:648] HybridClock initialized: now 1790651894417635 us; error 30 us; skew 500 ppm
Sep 29 03:18:14 dist-test-slave-nbjg krb5kdc[29627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651894, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:14.420306 30002 init.cc:377] Logged in from keytab as kudu/127.13.238.131@KRBTEST.COM (short username kudu)
I20260929 03:18:14.421118 30002 webserver.cc:460] Webserver started at http://127.13.238.131:36987/ using document root <none> and password file <none>
I20260929 03:18:14.421626 30002 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:14.421682 30002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:14.421841 30002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:18:14.423909 30002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/data/instance:
uuid: "d9d64153acee49f7bc033648fb6afac7"
format_stamp: "Formatted at 2026-09-29 03:18:14 on dist-test-slave-nbjg"
server_key: "054d5b7090832269ea0e72551efab345"
server_key_iv: "2929b683a1ecd0d0804bff2130a025b1"
server_key_version: "encryptionkey@0"
I20260929 03:18:14.424428 30002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance:
uuid: "d9d64153acee49f7bc033648fb6afac7"
format_stamp: "Formatted at 2026-09-29 03:18:14 on dist-test-slave-nbjg"
server_key: "054d5b7090832269ea0e72551efab345"
server_key_iv: "2929b683a1ecd0d0804bff2130a025b1"
server_key_version: "encryptionkey@0"
I20260929 03:18:14.428202 30002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 03:18:14.430883 30017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:14.432176 30002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:18:14.432277 30002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "d9d64153acee49f7bc033648fb6afac7"
format_stamp: "Formatted at 2026-09-29 03:18:14 on dist-test-slave-nbjg"
server_key: "054d5b7090832269ea0e72551efab345"
server_key_iv: "2929b683a1ecd0d0804bff2130a025b1"
server_key_version: "encryptionkey@0"
I20260929 03:18:14.432366 30002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:14.449627 30002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:14.452735 30002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:14.452891 30002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:14.453433 30002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:18:14.454258 30002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:18:14.454301 30002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:14.454340 30002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:18:14.454360 30002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:14.462561 30002 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:41419
I20260929 03:18:14.462603 30130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:41419 every 8 connection(s)
I20260929 03:18:14.463438 30002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:18:14.470950 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 30002
I20260929 03:18:14.471089 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance
I20260929 03:18:14.471339 14266 external_mini_cluster.cc:1428] Setting key 2f67715abaa90843c024587f34d0996f
Sep 29 03:18:14 dist-test-slave-nbjg krb5kdc[29627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651894, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:18:14.476290 29711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:14.465175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:43745 (local address 127.13.238.190:42831)
0929 03:18:14.465306 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:14.465309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:14.466394 (+ 1085us) server_negotiation.cc:407] Connection header received
0929 03:18:14.467311 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:14.467314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:14.467372 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:14.467447 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:14.468564 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.469100 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.470185 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.470444 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.470533 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:14.472997 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:14.473001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:14.473004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:14.473031 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:14.474329 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:14.474801 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:14.474804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:14.474805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:14.474919 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:14.475206 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:14.475208 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:14.475209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:14.475372 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:14.475470 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:14.475981 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 03:18:14.476130 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 03:18:14.477079 30133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:14.465554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.131:43745)
0929 03:18:14.466234 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:14.466282 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:14.467114 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:14.467625 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:14.467632 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:14.467964 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:14.468431 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:14.468445 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.469210 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.469214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:14.470074 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:14.470083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.470580 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.470583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:14.470743 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:14.471263 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:14.471279 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:14.472846 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:14.474445 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:14.474450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:14.474462 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:14.474710 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:14.475001 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:14.475003 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:14.475005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:14.475138 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:14.475500 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:14.475506 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:14.475730 (+ 224us) client_negotiation.cc:769] Sending connection context
0929 03:18:14.475895 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 03:18:14.476159 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":110,"threads_started":1}
I20260929 03:18:14.478176 30131 heartbeater.cc:347] Connected to a master server at 127.13.238.190:42831
I20260929 03:18:14.478440 30131 heartbeater.cc:464] Registering TS with master...
I20260929 03:18:14.478917 30131 heartbeater.cc:511] Master 127.13.238.190:42831 requested a full tablet report, sending...
I20260929 03:18:14.479882 29672 ts_manager.cc:194] Registered new tserver with Master: d9d64153acee49f7bc033648fb6afac7 (127.13.238.131:41419)
I20260929 03:18:14.480422 29672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.131@KRBTEST.COM'} at 127.13.238.131:43745
I20260929 03:18:14.485152 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:18:14.493654 29711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:14.487024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41396 (local address 127.13.238.190:42831)
0929 03:18:14.487175 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:14.487179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:14.487419 (+ 240us) server_negotiation.cc:407] Connection header received
0929 03:18:14.487532 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:14.487534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:14.487572 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:14.487634 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:14.488300 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.488786 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.489528 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.489734 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.489832 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:14.491084 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:14.491087 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:14.491089 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:14.491114 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:14.492254 (+ 1140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:14.492607 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:14.492610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:14.492611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:14.492682 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:14.492868 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:14.492871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:14.492872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:14.493097 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:14.493196 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:14.493448 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 03:18:14.493539 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:18:14.496500 29672 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:18:14.499550 29672 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:18:14.513819 30146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:14.508293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32916 (local address 127.13.238.130:42483)
0929 03:18:14.508772 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:14.508783 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:14.508807 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:18:14.508878 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:14.508883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:14.509080 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:14.509234 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:14.509634 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.510834 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.512154 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.513003 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.513129 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:14.513264 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:14.513555 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 03:18:14.513660 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":131,"threads_started":1}
I20260929 03:18:14.513826 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:14.507656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:42483 (local address 127.0.0.1:32916)
0929 03:18:14.508306 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:14.508319 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:14.508412 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:14.509196 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:14.509199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:14.509230 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:14.509496 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:14.509505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.510988 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.510991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:14.512034 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:14.512044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.513124 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.513128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:14.513330 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:14.513387 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 03:18:14.513525 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 03:18:14.513650 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":124,"spinlock_wait_cycles":6528,"thread_start_us":107,"threads_started":1}
I20260929 03:18:14.514302 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:14.508685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45078 (local address 127.13.238.129:43999)
0929 03:18:14.509070 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:14.509084 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:14.509105 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:18:14.509177 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:14.509182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:14.509347 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:14.509487 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:14.509833 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.510582 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.512255 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.513002 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.513129 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:14.513264 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:14.514029 (+ 765us) server_negotiation.cc:299] Negotiation successful
0929 03:18:14.514138 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":73,"threads_started":1}
I20260929 03:18:14.513831 30145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:14.507884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:43999 (local address 127.0.0.1:45078)
0929 03:18:14.508355 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:14.508366 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:14.508432 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:14.509635 (+ 1203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:14.509637 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:14.509654 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:14.509729 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:14.509733 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.510699 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.510702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:14.512128 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:14.512139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.513124 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.513128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:14.513326 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:14.513388 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 03:18:14.513497 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 03:18:14.513650 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":158,"thread_start_us":83,"threads_started":1}
I20260929 03:18:14.514611 30142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:14.507520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:41419 (local address 127.0.0.1:39188)
0929 03:18:14.508179 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:14.508228 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:14.508353 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:14.509188 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:14.509194 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:14.509232 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:14.509528 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:14.509537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.510967 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.510971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:14.512134 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:14.512141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.514179 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.514182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:14.514299 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:14.514322 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:18:14.514403 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 03:18:14.514472 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":94,"spinlock_wait_cycles":6528,"thread_start_us":69,"threads_started":1}
I20260929 03:18:14.515389 30144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:14.507697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39188 (local address 127.13.238.131:41419)
0929 03:18:14.508659 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:14.508671 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:14.508693 (+ 22us) server_negotiation.cc:407] Connection header received
0929 03:18:14.508769 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:14.508775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:14.508994 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:14.509452 (+ 458us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:14.509634 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.510834 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.513327 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.514077 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.514214 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:14.514339 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:14.515159 (+ 820us) server_negotiation.cc:299] Negotiation successful
0929 03:18:14.515256 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":863,"thread_start_us":126,"threads_started":1}
I20260929 03:18:14.517236 29797 tablet_service.cc:1467] Processing CreateTablet for tablet 56742b664e1747f3a7d9c6e44ca66d83 (DEFAULT_TABLE table=test-table [id=0e2dcbb09efe4ccdbbb8cfbbd657c46c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:14.517258 29931 tablet_service.cc:1467] Processing CreateTablet for tablet 56742b664e1747f3a7d9c6e44ca66d83 (DEFAULT_TABLE table=test-table [id=0e2dcbb09efe4ccdbbb8cfbbd657c46c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:14.517781 30065 tablet_service.cc:1467] Processing CreateTablet for tablet 56742b664e1747f3a7d9c6e44ca66d83 (DEFAULT_TABLE table=test-table [id=0e2dcbb09efe4ccdbbb8cfbbd657c46c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:14.518258 29797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56742b664e1747f3a7d9c6e44ca66d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:14.518347 29931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56742b664e1747f3a7d9c6e44ca66d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:14.518736 30065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56742b664e1747f3a7d9c6e44ca66d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:14.524107 30148 tablet_bootstrap.cc:492] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0: Bootstrap starting.
I20260929 03:18:14.525700 30149 tablet_bootstrap.cc:492] T 56742b664e1747f3a7d9c6e44ca66d83 P d9d64153acee49f7bc033648fb6afac7: Bootstrap starting.
I20260929 03:18:14.526079 30150 tablet_bootstrap.cc:492] T 56742b664e1747f3a7d9c6e44ca66d83 P 2a75ce578b5e4782980fdb5857faaf58: Bootstrap starting.
I20260929 03:18:14.526407 30148 tablet_bootstrap.cc:654] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:14.527155 30148 log.cc:824] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0: Log is configured to *not* fsync() on all Append() calls
I20260929 03:18:14.528342 30150 tablet_bootstrap.cc:654] T 56742b664e1747f3a7d9c6e44ca66d83 P 2a75ce578b5e4782980fdb5857faaf58: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:14.528858 30148 tablet_bootstrap.cc:492] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0: No bootstrap required, opened a new log
I20260929 03:18:14.529038 30148 ts_tablet_manager.cc:1397] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 03:18:14.529049 30150 log.cc:824] T 56742b664e1747f3a7d9c6e44ca66d83 P 2a75ce578b5e4782980fdb5857faaf58: Log is configured to *not* fsync() on all Append() calls
I20260929 03:18:14.531090 30150 tablet_bootstrap.cc:492] T 56742b664e1747f3a7d9c6e44ca66d83 P 2a75ce578b5e4782980fdb5857faaf58: No bootstrap required, opened a new log
I20260929 03:18:14.531278 30150 ts_tablet_manager.cc:1397] T 56742b664e1747f3a7d9c6e44ca66d83 P 2a75ce578b5e4782980fdb5857faaf58: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 03:18:14.531867 30149 tablet_bootstrap.cc:654] T 56742b664e1747f3a7d9c6e44ca66d83 P d9d64153acee49f7bc033648fb6afac7: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:14.532692 30149 log.cc:824] T 56742b664e1747f3a7d9c6e44ca66d83 P d9d64153acee49f7bc033648fb6afac7: Log is configured to *not* fsync() on all Append() calls
I20260929 03:18:14.532799 30148 raft_consensus.cc:348] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
I20260929 03:18:14.533063 30148 raft_consensus.cc:374] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:14.533136 30148 raft_consensus.cc:729] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb3a7ddf42d54d05a6638a441d6c48e0, State: Initialized, Role: FOLLOWER
I20260929 03:18:14.533565 30148 consensus_queue.cc:260] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
I20260929 03:18:14.534267 29863 heartbeater.cc:503] Master 127.13.238.190:42831 was elected leader, sending a full tablet report...
I20260929 03:18:14.534514 30148 ts_tablet_manager.cc:1428] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 03:18:14.534814 30150 raft_consensus.cc:348] T 56742b664e1747f3a7d9c6e44ca66d83 P 2a75ce578b5e4782980fdb5857faaf58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
I20260929 03:18:14.535077 30150 raft_consensus.cc:374] T 56742b664e1747f3a7d9c6e44ca66d83 P 2a75ce578b5e4782980fdb5857faaf58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:14.535137 30150 raft_consensus.cc:729] T 56742b664e1747f3a7d9c6e44ca66d83 P 2a75ce578b5e4782980fdb5857faaf58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a75ce578b5e4782980fdb5857faaf58, State: Initialized, Role: FOLLOWER
I20260929 03:18:14.535575 30150 consensus_queue.cc:260] T 56742b664e1747f3a7d9c6e44ca66d83 P 2a75ce578b5e4782980fdb5857faaf58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
I20260929 03:18:14.536336 29997 heartbeater.cc:503] Master 127.13.238.190:42831 was elected leader, sending a full tablet report...
I20260929 03:18:14.536373 30149 tablet_bootstrap.cc:492] T 56742b664e1747f3a7d9c6e44ca66d83 P d9d64153acee49f7bc033648fb6afac7: No bootstrap required, opened a new log
I20260929 03:18:14.536553 30149 ts_tablet_manager.cc:1397] T 56742b664e1747f3a7d9c6e44ca66d83 P d9d64153acee49f7bc033648fb6afac7: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260929 03:18:14.536577 30150 ts_tablet_manager.cc:1428] T 56742b664e1747f3a7d9c6e44ca66d83 P 2a75ce578b5e4782980fdb5857faaf58: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 03:18:14.539856 30149 raft_consensus.cc:348] T 56742b664e1747f3a7d9c6e44ca66d83 P d9d64153acee49f7bc033648fb6afac7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
I20260929 03:18:14.540083 30149 raft_consensus.cc:374] T 56742b664e1747f3a7d9c6e44ca66d83 P d9d64153acee49f7bc033648fb6afac7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:14.540122 30149 raft_consensus.cc:729] T 56742b664e1747f3a7d9c6e44ca66d83 P d9d64153acee49f7bc033648fb6afac7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d64153acee49f7bc033648fb6afac7, State: Initialized, Role: FOLLOWER
I20260929 03:18:14.540426 30149 consensus_queue.cc:260] T 56742b664e1747f3a7d9c6e44ca66d83 P d9d64153acee49f7bc033648fb6afac7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
I20260929 03:18:14.540963 30131 heartbeater.cc:503] Master 127.13.238.190:42831 was elected leader, sending a full tablet report...
I20260929 03:18:14.541240 30149 ts_tablet_manager.cc:1428] T 56742b664e1747f3a7d9c6e44ca66d83 P d9d64153acee49f7bc033648fb6afac7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 03:18:14.576467 30154 raft_consensus.cc:482] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:18:14.576678 30154 raft_consensus.cc:504] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
I20260929 03:18:14.577903 30154 leader_election.cc:288] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9d64153acee49f7bc033648fb6afac7 (127.13.238.131:41419), 2a75ce578b5e4782980fdb5857faaf58 (127.13.238.130:42483)
I20260929 03:18:14.583168 30144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:14.578190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:43183 (local address 127.13.238.131:41419)
0929 03:18:14.578356 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:14.578360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:14.578620 (+ 260us) server_negotiation.cc:407] Connection header received
0929 03:18:14.578798 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:14.578801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:14.578846 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:14.578915 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:14.579238 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.580051 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.581857 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.582497 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.582614 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:14.582666 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:14.582944 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 03:18:14.583070 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:18:14.583153 30146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:14.578276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:49801 (local address 127.13.238.130:42483)
0929 03:18:14.578401 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:14.578406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:14.578654 (+ 248us) server_negotiation.cc:407] Connection header received
0929 03:18:14.578780 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:14.578783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:14.578836 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:14.578916 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:14.579238 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.580023 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.581857 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.582497 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.582614 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:14.582666 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:14.582944 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 03:18:14.583029 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 03:18:14.583165 30158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:14.578168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:42483 (local address 127.13.238.129:49801)
0929 03:18:14.578580 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:14.578591 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:14.578683 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:14.578979 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:14.578982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:14.578994 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:14.579132 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:14.579138 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.580178 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.580181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:14.581737 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:14.581745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.582620 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.582624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:14.582763 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:14.582801 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 03:18:14.582888 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 03:18:14.582975 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":128,"spinlock_wait_cycles":2944,"thread_start_us":116,"threads_started":1}
I20260929 03:18:14.583164 30157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:14.578117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:41419 (local address 127.13.238.129:43183)
0929 03:18:14.578520 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:14.578535 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:14.578683 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:14.578935 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:14.578937 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:14.578955 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:14.579131 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:14.579138 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.580172 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.580175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:14.581733 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:14.581745 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.582620 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.582624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:14.582767 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:14.582794 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 03:18:14.582888 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 03:18:14.582993 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":207,"spinlock_wait_cycles":14976,"thread_start_us":141,"threads_started":1}
I20260929 03:18:14.583902 29951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56742b664e1747f3a7d9c6e44ca66d83" candidate_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a75ce578b5e4782980fdb5857faaf58" is_pre_election: true
I20260929 03:18:14.583899 30085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56742b664e1747f3a7d9c6e44ca66d83" candidate_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d64153acee49f7bc033648fb6afac7" is_pre_election: true
I20260929 03:18:14.584177 29951 raft_consensus.cc:2454] T 56742b664e1747f3a7d9c6e44ca66d83 P 2a75ce578b5e4782980fdb5857faaf58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb3a7ddf42d54d05a6638a441d6c48e0 in term 0.
I20260929 03:18:14.584184 30085 raft_consensus.cc:2454] T 56742b664e1747f3a7d9c6e44ca66d83 P d9d64153acee49f7bc033648fb6afac7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb3a7ddf42d54d05a6638a441d6c48e0 in term 0.
I20260929 03:18:14.584614 29753 leader_election.cc:302] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9d64153acee49f7bc033648fb6afac7, eb3a7ddf42d54d05a6638a441d6c48e0; no voters:
I20260929 03:18:14.584872 30154 raft_consensus.cc:2781] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:18:14.584961 30154 raft_consensus.cc:482] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:18:14.585011 30154 raft_consensus.cc:3037] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:14.586122 30154 raft_consensus.cc:504] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
I20260929 03:18:14.586495 30154 leader_election.cc:288] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0 [CANDIDATE]: Term 1 election: Requested vote from peers d9d64153acee49f7bc033648fb6afac7 (127.13.238.131:41419), 2a75ce578b5e4782980fdb5857faaf58 (127.13.238.130:42483)
I20260929 03:18:14.586849 30085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56742b664e1747f3a7d9c6e44ca66d83" candidate_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d64153acee49f7bc033648fb6afac7"
I20260929 03:18:14.586877 29951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56742b664e1747f3a7d9c6e44ca66d83" candidate_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a75ce578b5e4782980fdb5857faaf58"
I20260929 03:18:14.586970 30085 raft_consensus.cc:3037] T 56742b664e1747f3a7d9c6e44ca66d83 P d9d64153acee49f7bc033648fb6afac7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:14.587019 29951 raft_consensus.cc:3037] T 56742b664e1747f3a7d9c6e44ca66d83 P 2a75ce578b5e4782980fdb5857faaf58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:14.587988 29951 raft_consensus.cc:2454] T 56742b664e1747f3a7d9c6e44ca66d83 P 2a75ce578b5e4782980fdb5857faaf58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb3a7ddf42d54d05a6638a441d6c48e0 in term 1.
I20260929 03:18:14.587991 30085 raft_consensus.cc:2454] T 56742b664e1747f3a7d9c6e44ca66d83 P d9d64153acee49f7bc033648fb6afac7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb3a7ddf42d54d05a6638a441d6c48e0 in term 1.
I20260929 03:18:14.588255 29753 leader_election.cc:302] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9d64153acee49f7bc033648fb6afac7, eb3a7ddf42d54d05a6638a441d6c48e0; no voters:
I20260929 03:18:14.588421 30154 raft_consensus.cc:2781] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:18:14.588625 30154 raft_consensus.cc:686] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 LEADER]: Becoming Leader. State: Replica: eb3a7ddf42d54d05a6638a441d6c48e0, State: Running, Role: LEADER
I20260929 03:18:14.588927 30154 consensus_queue.cc:237] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
I20260929 03:18:14.592258 29672 catalog_manager.cc:5495] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb3a7ddf42d54d05a6638a441d6c48e0 (127.13.238.129). New cstate: current_term: 1 leader_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } health_report { overall_health: HEALTHY } } }
I20260929 03:18:14.614027 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:14.610609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45086 (local address 127.13.238.129:43999)
0929 03:18:14.610733 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:14.610737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:14.610774 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:18:14.610849 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:14.610851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:14.610908 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:14.610985 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:18:14.611218 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.611744 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.612610 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:14.612826 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:14.612913 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:14.613230 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:18:14.613650 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:18:14.613755 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:14.613829 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 03:18:14.613887 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:18:14.619706 30085 raft_consensus.cc:1261] T 56742b664e1747f3a7d9c6e44ca66d83 P d9d64153acee49f7bc033648fb6afac7 [term 1 FOLLOWER]: Refusing update from remote peer eb3a7ddf42d54d05a6638a441d6c48e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:18:14.619705 29951 raft_consensus.cc:1261] T 56742b664e1747f3a7d9c6e44ca66d83 P 2a75ce578b5e4782980fdb5857faaf58 [term 1 FOLLOWER]: Refusing update from remote peer eb3a7ddf42d54d05a6638a441d6c48e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:18:14.620381 30159 consensus_queue.cc:1035] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:18:14.620564 30154 consensus_queue.cc:1035] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:18:14.630110 30163 mvcc.cc:205] Tried to move back new op lower bound from 7334510160355418112 to 7334510160239017984. Current Snapshot: MvccSnapshot[applied={T|T < 7334510160355418112}]
I20260929 03:18:14.636983 29672 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:18:14.637145 29672 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:18:14.639554 29672 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce: Sending DeleteTablet for 3 replicas of tablet 56742b664e1747f3a7d9c6e44ca66d83
I20260929 03:18:14.640398 30065 tablet_service.cc:1514] Processing DeleteTablet for tablet 56742b664e1747f3a7d9c6e44ca66d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:14 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:39188
I20260929 03:18:14.640398 29931 tablet_service.cc:1514] Processing DeleteTablet for tablet 56742b664e1747f3a7d9c6e44ca66d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:14 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:32916
I20260929 03:18:14.640527 29797 tablet_service.cc:1514] Processing DeleteTablet for tablet 56742b664e1747f3a7d9c6e44ca66d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:14 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:45078
I20260929 03:18:14.641004 30174 tablet_replica.cc:330] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0: stopping tablet replica
I20260929 03:18:14.641077 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 29734
I20260929 03:18:14.641185 30173 tablet_replica.cc:330] T 56742b664e1747f3a7d9c6e44ca66d83 P d9d64153acee49f7bc033648fb6afac7: stopping tablet replica
I20260929 03:18:14.641356 30174 raft_consensus.cc:2229] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:18:14.641402 30173 raft_consensus.cc:2229] T 56742b664e1747f3a7d9c6e44ca66d83 P d9d64153acee49f7bc033648fb6afac7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:18:14.641747 30173 raft_consensus.cc:2258] T 56742b664e1747f3a7d9c6e44ca66d83 P d9d64153acee49f7bc033648fb6afac7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:18:14.642066 30172 tablet_replica.cc:330] T 56742b664e1747f3a7d9c6e44ca66d83 P 2a75ce578b5e4782980fdb5857faaf58: stopping tablet replica
I20260929 03:18:14.642379 30172 raft_consensus.cc:2229] T 56742b664e1747f3a7d9c6e44ca66d83 P 2a75ce578b5e4782980fdb5857faaf58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:18:14.642673 30172 raft_consensus.cc:2258] T 56742b664e1747f3a7d9c6e44ca66d83 P 2a75ce578b5e4782980fdb5857faaf58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:18:14.643347 30173 ts_tablet_manager.cc:1905] T 56742b664e1747f3a7d9c6e44ca66d83 P d9d64153acee49f7bc033648fb6afac7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:18:14.643955 30172 ts_tablet_manager.cc:1905] T 56742b664e1747f3a7d9c6e44ca66d83 P 2a75ce578b5e4782980fdb5857faaf58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:18:14.646068 30173 ts_tablet_manager.cc:1918] T 56742b664e1747f3a7d9c6e44ca66d83 P d9d64153acee49f7bc033648fb6afac7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:18:14.646167 30173 log.cc:1196] T 56742b664e1747f3a7d9c6e44ca66d83 P d9d64153acee49f7bc033648fb6afac7: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/56742b664e1747f3a7d9c6e44ca66d83
I20260929 03:18:14.646556 30173 ts_tablet_manager.cc:1939] T 56742b664e1747f3a7d9c6e44ca66d83 P d9d64153acee49f7bc033648fb6afac7: Deleting consensus metadata
I20260929 03:18:14.646536 30172 ts_tablet_manager.cc:1918] T 56742b664e1747f3a7d9c6e44ca66d83 P 2a75ce578b5e4782980fdb5857faaf58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:18:14.646627 30172 log.cc:1196] T 56742b664e1747f3a7d9c6e44ca66d83 P 2a75ce578b5e4782980fdb5857faaf58: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/wal/wals/56742b664e1747f3a7d9c6e44ca66d83
I20260929 03:18:14.646948 30172 ts_tablet_manager.cc:1939] T 56742b664e1747f3a7d9c6e44ca66d83 P 2a75ce578b5e4782980fdb5857faaf58: Deleting consensus metadata
I20260929 03:18:14.647908 29659 catalog_manager.cc:4841] TS d9d64153acee49f7bc033648fb6afac7 (127.13.238.131:41419): tablet 56742b664e1747f3a7d9c6e44ca66d83 (table test-table [id=0e2dcbb09efe4ccdbbb8cfbbd657c46c]) successfully deleted
W20260929 03:18:14.648149 29657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.238.129:43999 (error 108)
I20260929 03:18:14.648456 29658 catalog_manager.cc:4841] TS 2a75ce578b5e4782980fdb5857faaf58 (127.13.238.130:42483): tablet 56742b664e1747f3a7d9c6e44ca66d83 (table test-table [id=0e2dcbb09efe4ccdbbb8cfbbd657c46c]) successfully deleted
I20260929 03:18:14.648922 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 29868
I20260929 03:18:14.649070 30142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:14.648813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:43999 (local address 127.0.0.1:45100)
0929 03:18:14.648928 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:14.648983 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.129:43999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 03:18:14.649528 29657 catalog_manager.cc:4584] TS eb3a7ddf42d54d05a6638a441d6c48e0 (127.13.238.129:43999): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 56742b664e1747f3a7d9c6e44ca66d83: Network error: Client connection negotiation failed: client connection to 127.13.238.129:43999: connect: Connection refused (error 111)
I20260929 03:18:14.654326 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 30002
I20260929 03:18:14.659162 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 29640
I20260929 03:18:34.664757 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:42831
--webserver_interface=127.13.238.190
--webserver_port=33565
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:42831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:18:34.775132 30176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:34.775388 30176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:34.775425 30176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:34.779397 30176 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 03:18:34.779446 30176 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 03:18:34.779461 30176 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 03:18:34.779472 30176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:18:34.779488 30176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:34.779505 30176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:34.779518 30176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:18:34.779538 30176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:18:34.784772 30176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:42831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:42831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=33565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:18:34.785790 30176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:34.786860 30176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:34.800064 30185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:34.800107 30182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:34.800094 30183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:34.800235 30176 server_base.cc:1034] running on GCE node
I20260929 03:18:34.800890 30176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:34.802032 30176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:34.803193 30176 hybrid_clock.cc:648] HybridClock initialized: now 1790651914803176 us; error 54 us; skew 500 ppm
Sep 29 03:18:34 dist-test-slave-nbjg krb5kdc[29627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651914, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:34.808769 30176 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:18:34.809705 30176 webserver.cc:460] Webserver started at http://127.13.238.190:33565/ using document root <none> and password file <none>
I20260929 03:18:34.810300 30176 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:34.810379 30176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:34.814394 30176 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:18:34.816547 30192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:34.817896 30176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:18:34.818022 30176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "6a674960548b41ef9bdd6508f2e421ce"
format_stamp: "Formatted at 2026-09-29 03:18:13 on dist-test-slave-nbjg"
server_key: "491f9449b5027cd553d1083c533c5607"
server_key_iv: "01ca876096e726f385b02b4b205811ce"
server_key_version: "encryptionkey@0"
I20260929 03:18:34.818660 30176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:34.834976 30176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:34.838297 30176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:34.838479 30176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:34.845522 30176 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:42831
I20260929 03:18:34.846158 30244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:42831 every 8 connection(s)
I20260929 03:18:34.847402 30176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:18:34.851306 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 30176
I20260929 03:18:34.851824 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:43999
--local_ip_for_outbound_sockets=127.13.238.129
--tserver_master_addrs=127.13.238.190:42831
--webserver_port=45617
--webserver_interface=127.13.238.129
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:18:34.852339 30245 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:18:34.854450 30245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce: Bootstrap starting.
I20260929 03:18:34.872596 30245 log.cc:824] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce: Log is configured to *not* fsync() on all Append() calls
I20260929 03:18:34.881682 30245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:18:34.882035 30245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce: Bootstrap complete.
I20260929 03:18:34.886001 30245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a674960548b41ef9bdd6508f2e421ce" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42831 } }
I20260929 03:18:34.886431 30245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a674960548b41ef9bdd6508f2e421ce, State: Initialized, Role: FOLLOWER
I20260929 03:18:34.886922 30245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a674960548b41ef9bdd6508f2e421ce" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42831 } }
I20260929 03:18:34.887046 30245 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:18:34.887115 30245 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:18:34.887218 30245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:18:34.889098 30245 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a674960548b41ef9bdd6508f2e421ce" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42831 } }
I20260929 03:18:34.889429 30245 leader_election.cc:302] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a674960548b41ef9bdd6508f2e421ce; no voters:
I20260929 03:18:34.889726 30245 leader_election.cc:288] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 03:18:34.889832 30249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:18:34.890100 30249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [term 2 LEADER]: Becoming Leader. State: Replica: 6a674960548b41ef9bdd6508f2e421ce, State: Running, Role: LEADER
I20260929 03:18:34.890489 30249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a674960548b41ef9bdd6508f2e421ce" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42831 } }
I20260929 03:18:34.891036 30245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:18:34.891961 30251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a674960548b41ef9bdd6508f2e421ce. Latest consensus state: current_term: 2 leader_uuid: "6a674960548b41ef9bdd6508f2e421ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a674960548b41ef9bdd6508f2e421ce" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42831 } } }
I20260929 03:18:34.892010 30250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6a674960548b41ef9bdd6508f2e421ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a674960548b41ef9bdd6508f2e421ce" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42831 } } }
I20260929 03:18:34.892100 30251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [sys.catalog]: This master's current role is: LEADER
I20260929 03:18:34.892117 30250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [sys.catalog]: This master's current role is: LEADER
I20260929 03:18:34.892452 30258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:18:34.897382 30258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:18:34.898319 30258 catalog_manager.cc:1259] Loaded cluster ID: 3dd2aa39bee746e5aea28c03060e1e39
I20260929 03:18:34.898424 30258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:18:34.901081 30258 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:18:34.901937 30258 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce: Loaded TSK: 0
I20260929 03:18:34.914288 30258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce: Generated new TSK 1
I20260929 03:18:34.914546 30258 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:18:34.970719 30247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:34.970973 30247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:34.971012 30247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:34.975025 30247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:34.975082 30247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:34.975142 30247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:18:34.980347 30247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:43999
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=45617
--tserver_master_addrs=127.13.238.190:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:18:34.981338 30247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:34.982410 30247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:34.991947 30273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:34.992038 30272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:34.992162 30247 server_base.cc:1034] running on GCE node
W20260929 03:18:34.992302 30275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:34.992699 30247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:34.993387 30247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:34.994555 30247 hybrid_clock.cc:648] HybridClock initialized: now 1790651914994523 us; error 47 us; skew 500 ppm
Sep 29 03:18:34 dist-test-slave-nbjg krb5kdc[29627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651914, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:34.997674 30247 init.cc:377] Logged in from keytab as kudu/127.13.238.129@KRBTEST.COM (short username kudu)
I20260929 03:18:34.998840 30247 webserver.cc:460] Webserver started at http://127.13.238.129:45617/ using document root <none> and password file <none>
I20260929 03:18:34.999565 30247 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:34.999680 30247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:35.004998 30247 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 03:18:35.007730 30282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:35.009258 30247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:18:35.009382 30247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "eb3a7ddf42d54d05a6638a441d6c48e0"
format_stamp: "Formatted at 2026-09-29 03:18:13 on dist-test-slave-nbjg"
server_key: "447755d56fbd7aa9d1da9953ed2addfd"
server_key_iv: "eef7261c1439168672bdbbb926395303"
server_key_version: "encryptionkey@0"
I20260929 03:18:35.010095 30247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:35.036682 30247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:35.040268 30247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:35.040477 30247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:35.041177 30247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:18:35.042481 30289 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 03:18:35.046617 30247 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 03:18:35.046690 30247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 03:18:35.046815 30247 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 03:18:35.048460 30247 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 03:18:35.048503 30247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 03:18:35.048813 30289 tablet_bootstrap.cc:492] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0: Bootstrap starting.
I20260929 03:18:35.058068 30247 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:43999
I20260929 03:18:35.058125 30396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:43999 every 8 connection(s)
I20260929 03:18:35.059041 30247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:18:35.063100 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 30247
I20260929 03:18:35.063509 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:42483
--local_ip_for_outbound_sockets=127.13.238.130
--tserver_master_addrs=127.13.238.190:42831
--webserver_port=37135
--webserver_interface=127.13.238.130
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:18:35.067332 30289 log.cc:824] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0: Log is configured to *not* fsync() on all Append() calls
Sep 29 03:18:35 dist-test-slave-nbjg krb5kdc[29627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651914, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:18:35.075294 30289 tablet_bootstrap.cc:492] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:18:35.075627 30289 tablet_bootstrap.cc:492] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0: Bootstrap complete.
I20260929 03:18:35.076895 30289 ts_tablet_manager.cc:1397] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0: Time spent bootstrapping tablet: real 0.028s user 0.019s sys 0.008s
I20260929 03:18:35.077459 30400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:35.061053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:59389 (local address 127.13.238.190:42831)
0929 03:18:35.061576 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:35.061594 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:35.061641 (+ 47us) server_negotiation.cc:407] Connection header received
0929 03:18:35.062873 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:35.062889 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:35.063331 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:35.063791 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:35.065439 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.066547 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.068059 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.068642 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.068805 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:35.071826 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:35.071834 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:35.071852 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:35.071889 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:35.073992 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:35.074606 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:35.074613 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:35.074619 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:35.074779 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:35.075148 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:35.075151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:35.075152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:35.075586 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:35.075710 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:35.076333 (+ 623us) server_negotiation.cc:299] Negotiation successful
0929 03:18:35.076629 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":120,"threads_started":1}
I20260929 03:18:35.077472 30399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:35.060996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.129:59389)
0929 03:18:35.061481 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:35.061534 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:35.062387 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:35.064098 (+ 1711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:35.064106 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:35.064582 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:35.065265 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:35.065281 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.066689 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.066692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:35.067934 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:35.067945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.068800 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.068805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:35.069014 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:35.069698 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:35.069720 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:35.071597 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:35.074122 (+ 2525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:35.074128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:35.074141 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:35.074492 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:35.074887 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:35.074890 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:35.074892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:35.075056 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:35.075726 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:35.075733 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:35.076015 (+ 282us) client_negotiation.cc:769] Sending connection context
0929 03:18:35.076187 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 03:18:35.076492 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":108,"threads_started":1}
I20260929 03:18:35.079584 30289 raft_consensus.cc:348] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
I20260929 03:18:35.080080 30289 raft_consensus.cc:729] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb3a7ddf42d54d05a6638a441d6c48e0, State: Initialized, Role: FOLLOWER
I20260929 03:18:35.080572 30289 consensus_queue.cc:260] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
I20260929 03:18:35.080704 30397 heartbeater.cc:347] Connected to a master server at 127.13.238.190:42831
I20260929 03:18:35.080950 30397 heartbeater.cc:464] Registering TS with master...
I20260929 03:18:35.081430 30289 ts_tablet_manager.cc:1428] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260929 03:18:35.081517 30397 heartbeater.cc:511] Master 127.13.238.190:42831 requested a full tablet report, sending...
I20260929 03:18:35.084161 30209 ts_manager.cc:194] Registered new tserver with Master: eb3a7ddf42d54d05a6638a441d6c48e0 (127.13.238.129:43999)
I20260929 03:18:35.088182 30209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.129@KRBTEST.COM'} at 127.13.238.129:59389
I20260929 03:18:35.091210 30397 heartbeater.cc:503] Master 127.13.238.190:42831 was elected leader, sending a full tablet report...
Sep 29 03:18:35 dist-test-slave-nbjg krb5kdc[29627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651914, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for kudu/127.13.238.129@KRBTEST.COM
I20260929 03:18:35.097774 30404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:35.087468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:43999 (local address 127.0.0.1:49122)
0929 03:18:35.088018 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:35.088063 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:35.088197 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:35.088658 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:35.088664 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:35.089040 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:35.089174 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:35.089182 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.090337 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.090341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:35.091015 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:35.091026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.091701 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.091705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:35.091814 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:35.092317 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:35.092329 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:35.093937 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:35.095935 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:35.095945 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:35.095950 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:35.096242 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:35.096532 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:35.096535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:35.096537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:35.096632 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:35.097132 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:35.097137 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:35.097271 (+ 134us) client_negotiation.cc:769] Sending connection context
0929 03:18:35.097413 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 03:18:35.097606 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":128,"threads_started":1}
I20260929 03:18:35.097787 30405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:35.087544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49122 (local address 127.13.238.129:43999)
0929 03:18:35.088252 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:35.088263 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:35.088298 (+ 35us) server_negotiation.cc:407] Connection header received
0929 03:18:35.088360 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:35.088364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:35.088519 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:35.088634 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:35.089293 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.090164 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.091133 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.091584 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.091683 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:35.094098 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:35.094117 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:35.094131 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:35.094158 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:35.095824 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:35.096364 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:35.096369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:35.096371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:35.096451 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:35.096785 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:35.096788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:35.096790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:35.097032 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:35.097131 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:35.097473 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 03:18:35.097609 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":614,"thread_start_us":78,"threads_started":1}
I20260929 03:18:35.098878 30331 tablet_service.cc:1514] Processing DeleteTablet for tablet 56742b664e1747f3a7d9c6e44ca66d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:14 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:49122
I20260929 03:18:35.099915 30406 tablet_replica.cc:330] stopping tablet replica
I20260929 03:18:35.100203 30406 raft_consensus.cc:2229] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:18:35.100449 30406 raft_consensus.cc:2258] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:18:35.101361 30406 ts_tablet_manager.cc:1905] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:18:35.104882 30406 ts_tablet_manager.cc:1918] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:18:35.104969 30406 log.cc:1196] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/wal/wals/56742b664e1747f3a7d9c6e44ca66d83
I20260929 03:18:35.105181 30406 ts_tablet_manager.cc:1939] T 56742b664e1747f3a7d9c6e44ca66d83 P eb3a7ddf42d54d05a6638a441d6c48e0: Deleting consensus metadata
I20260929 03:18:35.107168 30194 catalog_manager.cc:4841] TS eb3a7ddf42d54d05a6638a441d6c48e0 (127.13.238.129:43999): tablet 56742b664e1747f3a7d9c6e44ca66d83 (table test-table [id=0e2dcbb09efe4ccdbbb8cfbbd657c46c]) successfully deleted
W20260929 03:18:35.183014 30401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:35.183269 30401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:35.183308 30401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:35.187371 30401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:35.187435 30401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:35.187497 30401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:18:35.192927 30401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:42483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=37135
--tserver_master_addrs=127.13.238.190:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:18:35.193988 30401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:35.195050 30401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:35.205672 30414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:35.205854 30412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:35.205731 30411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:35.206527 30401 server_base.cc:1034] running on GCE node
I20260929 03:18:35.206988 30401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:35.207681 30401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:35.208822 30401 hybrid_clock.cc:648] HybridClock initialized: now 1790651915208751 us; error 83 us; skew 500 ppm
Sep 29 03:18:35 dist-test-slave-nbjg krb5kdc[29627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651915, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:35.211516 30401 init.cc:377] Logged in from keytab as kudu/127.13.238.130@KRBTEST.COM (short username kudu)
I20260929 03:18:35.212428 30401 webserver.cc:460] Webserver started at http://127.13.238.130:37135/ using document root <none> and password file <none>
I20260929 03:18:35.212955 30401 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:35.213025 30401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:35.216936 30401 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:18:35.218921 30421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:35.220099 30401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:18:35.220193 30401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "2a75ce578b5e4782980fdb5857faaf58"
format_stamp: "Formatted at 2026-09-29 03:18:14 on dist-test-slave-nbjg"
server_key: "c3fab53dc74dfa61b85710d05a7355e2"
server_key_iv: "77a035d42472b51565293e2b0fde7b1c"
server_key_version: "encryptionkey@0"
I20260929 03:18:35.220701 30401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:35.233412 30401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:35.236234 30401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:35.236395 30401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:35.236970 30401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:18:35.237746 30401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:18:35.237788 30401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:35.237826 30401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:18:35.237854 30401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:35.246332 30401 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:42483
I20260929 03:18:35.246361 30534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:42483 every 8 connection(s)
I20260929 03:18:35.247310 30401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:18:35.250599 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 30401
I20260929 03:18:35.251055 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:41419
--local_ip_for_outbound_sockets=127.13.238.131
--tserver_master_addrs=127.13.238.190:42831
--webserver_port=36987
--webserver_interface=127.13.238.131
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:18:35 dist-test-slave-nbjg krb5kdc[29627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651915, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:18:35.260885 30400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:35.249170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:52287 (local address 127.13.238.190:42831)
0929 03:18:35.249326 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:35.249331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:35.250428 (+ 1097us) server_negotiation.cc:407] Connection header received
0929 03:18:35.251611 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:35.251614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:35.251663 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:35.251738 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:35.253104 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.253640 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.254841 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.255079 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.255165 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:35.257555 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:35.257558 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:35.257560 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:35.257587 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:35.258917 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:35.259419 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:35.259422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:35.259423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:35.259501 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:35.259805 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:35.259809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:35.259810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:35.259991 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:35.260074 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:35.260559 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 03:18:35.260695 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:18:35.261686 30537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:35.249597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.130:52287)
0929 03:18:35.250261 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:35.250315 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:35.251354 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:35.251949 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:35.251958 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:35.252420 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:35.252960 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:35.252975 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.253756 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.253760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:35.254722 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:35.254732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.255217 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.255221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:35.255380 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:35.256003 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:35.256018 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:35.257388 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:35.259020 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:35.259026 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:35.259042 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:35.259290 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:35.259584 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:35.259587 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:35.259589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:35.259723 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:35.260084 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:35.260091 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:35.260336 (+ 245us) client_negotiation.cc:769] Sending connection context
0929 03:18:35.260502 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 03:18:35.260827 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":103,"threads_started":1}
I20260929 03:18:35.262858 30535 heartbeater.cc:347] Connected to a master server at 127.13.238.190:42831
I20260929 03:18:35.263085 30535 heartbeater.cc:464] Registering TS with master...
I20260929 03:18:35.263619 30535 heartbeater.cc:511] Master 127.13.238.190:42831 requested a full tablet report, sending...
I20260929 03:18:35.264739 30209 ts_manager.cc:194] Registered new tserver with Master: 2a75ce578b5e4782980fdb5857faaf58 (127.13.238.130:42483)
I20260929 03:18:35.265492 30209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.130@KRBTEST.COM'} at 127.13.238.130:52287
W20260929 03:18:35.361845 30538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:35.362097 30538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:35.362136 30538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:35.366071 30538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:35.366139 30538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:35.366217 30538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:18:35.371464 30538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:41419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=36987
--tserver_master_addrs=127.13.238.190:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:18:35.372460 30538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:35.373420 30538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:35.382774 30544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:35.382831 30546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:35.382939 30543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:35.383144 30538 server_base.cc:1034] running on GCE node
I20260929 03:18:35.383509 30538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:35.384097 30538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:35.385272 30538 hybrid_clock.cc:648] HybridClock initialized: now 1790651915385239 us; error 44 us; skew 500 ppm
Sep 29 03:18:35 dist-test-slave-nbjg krb5kdc[29627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651915, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:35.414589 30538 init.cc:377] Logged in from keytab as kudu/127.13.238.131@KRBTEST.COM (short username kudu)
I20260929 03:18:35.415710 30538 webserver.cc:460] Webserver started at http://127.13.238.131:36987/ using document root <none> and password file <none>
I20260929 03:18:35.416325 30538 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:35.416412 30538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:35.420475 30538 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:18:35.422629 30553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:35.423956 30538 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:18:35.424070 30538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "d9d64153acee49f7bc033648fb6afac7"
format_stamp: "Formatted at 2026-09-29 03:18:14 on dist-test-slave-nbjg"
server_key: "054d5b7090832269ea0e72551efab345"
server_key_iv: "2929b683a1ecd0d0804bff2130a025b1"
server_key_version: "encryptionkey@0"
I20260929 03:18:35.424660 30538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:35.445828 30538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:35.449018 30538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:35.449193 30538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:35.449796 30538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:18:35.450644 30538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:18:35.450691 30538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:35.450744 30538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:18:35.450773 30538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:35.459417 30538 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:41419
I20260929 03:18:35.459453 30666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:41419 every 8 connection(s)
I20260929 03:18:35.460445 30538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:18:35.468019 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 30538
Sep 29 03:18:35 dist-test-slave-nbjg krb5kdc[29627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651915, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:18:35.474362 30400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:35.462277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:38793 (local address 127.13.238.190:42831)
0929 03:18:35.462403 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:35.462407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:35.463480 (+ 1073us) server_negotiation.cc:407] Connection header received
0929 03:18:35.464382 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:35.464386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:35.464444 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:35.464516 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:35.465690 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.466256 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.467410 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.467630 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.467718 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:35.470419 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:35.470422 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:35.470424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:35.470448 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:35.472030 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:35.472582 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:35.472585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:35.472587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:35.472684 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:35.473174 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:35.473177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:35.473180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:35.473398 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:35.473505 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:35.474042 (+ 537us) server_negotiation.cc:299] Negotiation successful
0929 03:18:35.474165 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:18:35.475390 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:35.462669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.131:38793)
0929 03:18:35.463303 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:35.463341 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:35.464178 (+ 837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:35.464696 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:35.464704 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:35.465071 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:35.465548 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:35.465561 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.466365 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.466368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:35.467300 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:35.467309 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.467764 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.467768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:35.467931 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:35.468470 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:35.468487 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:35.470256 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:35.472133 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:35.472139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:35.472155 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:35.472491 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:35.472938 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:35.472940 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:35.472942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:35.473077 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:35.473489 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:35.473494 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:35.473735 (+ 241us) client_negotiation.cc:769] Sending connection context
0929 03:18:35.473983 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 03:18:35.474323 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":113,"threads_started":1}
I20260929 03:18:35.476583 30667 heartbeater.cc:347] Connected to a master server at 127.13.238.190:42831
I20260929 03:18:35.476835 30667 heartbeater.cc:464] Registering TS with master...
I20260929 03:18:35.477458 30671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:35.468660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44248 (local address 127.13.238.190:42831)
0929 03:18:35.468936 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:35.468940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:35.468956 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:18:35.469075 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:35.469078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:35.469127 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:35.469214 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:35.470094 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.470774 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.472034 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.472360 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.472468 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:35.473784 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:35.473788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:35.473790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:35.473813 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:35.475392 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:35.475888 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:35.475892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:35.475894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:35.475969 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:35.476614 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:35.476616 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:35.476617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:35.476822 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:35.477020 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:35.477235 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 03:18:35.477333 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":95,"threads_started":1}
I20260929 03:18:35.477463 30667 heartbeater.cc:511] Master 127.13.238.190:42831 requested a full tablet report, sending...
I20260929 03:18:35.478633 30209 ts_manager.cc:194] Registered new tserver with Master: d9d64153acee49f7bc033648fb6afac7 (127.13.238.131:41419)
I20260929 03:18:35.479269 30209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.131@KRBTEST.COM'} at 127.13.238.131:38793
I20260929 03:18:35.490957 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:18:35.495262 30671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:35.491593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44264 (local address 127.13.238.190:42831)
0929 03:18:35.491708 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:35.491711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:35.491945 (+ 234us) server_negotiation.cc:407] Connection header received
0929 03:18:35.492131 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:35.492134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:35.492247 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:35.492340 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:18:35.492688 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.493219 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.494154 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.494414 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.494493 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:35.494783 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:18:35.494956 (+ 173us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 03:18:35.495161 (+ 205us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44264: token expired
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 03:18:35.495334 30671 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44264: token expired
W20260929 03:18:35.495694 30137 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.13.238.190:42831)
I20260929 03:18:35.511708 30671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:35.496032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44266 (local address 127.13.238.190:42831)
0929 03:18:35.496135 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:35.496139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:35.496166 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:18:35.496284 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:35.496287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:35.496326 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:35.496386 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:35.496971 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.497436 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.499386 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.499828 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.499985 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:35.507916 (+ 7931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:35.507921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:35.507923 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:35.507959 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:35.509698 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:35.510188 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:35.510191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:35.510217 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:35.510310 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:35.510571 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:35.510575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:35.510576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:35.510800 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:35.510935 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:35.511151 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 03:18:35.511535 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:18:35.515046 30209 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:18:35.519598 30209 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:18:35.528692 30331 tablet_service.cc:1467] Processing CreateTablet for tablet 18156cbf5dc94bd2a4b98a1e521fc973 (DEFAULT_TABLE table=test-table [id=198bf811f6624bd5b967fd69ffb53c1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:35.529387 30331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18156cbf5dc94bd2a4b98a1e521fc973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:35.533010 30289 tablet_bootstrap.cc:492] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0: Bootstrap starting.
I20260929 03:18:35.533165 30404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:35.527169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:41419 (local address 127.0.0.1:55904)
0929 03:18:35.527299 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:35.527314 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:35.527399 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:35.528267 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:35.528271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:35.528290 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:35.528400 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:35.528407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.530038 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.530041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:35.531452 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:35.531465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.532689 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.532694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:35.532794 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:35.532861 (+ 67us) client_negotiation.cc:769] Sending connection context
0929 03:18:35.532942 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 03:18:35.533036 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 03:18:35.533524 30676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:35.527390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55904 (local address 127.13.238.131:41419)
0929 03:18:35.527719 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:35.527732 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:35.527758 (+ 26us) server_negotiation.cc:407] Connection header received
0929 03:18:35.527855 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:35.527862 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:35.528090 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:35.528655 (+ 565us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:35.528831 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.529912 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.531612 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.532525 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.532695 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:35.532865 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:35.532983 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 03:18:35.533203 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":110,"threads_started":1}
I20260929 03:18:35.533980 30289 tablet_bootstrap.cc:654] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:35.535885 30289 tablet_bootstrap.cc:492] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0: No bootstrap required, opened a new log
I20260929 03:18:35.535977 30289 ts_tablet_manager.cc:1397] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:18:35.536435 30289 raft_consensus.cc:348] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
I20260929 03:18:35.536563 30289 raft_consensus.cc:374] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:35.536602 30289 raft_consensus.cc:729] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb3a7ddf42d54d05a6638a441d6c48e0, State: Initialized, Role: FOLLOWER
I20260929 03:18:35.536911 30289 consensus_queue.cc:260] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
I20260929 03:18:35.537665 30289 ts_tablet_manager.cc:1428] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:18:35.537254 30601 tablet_service.cc:1467] Processing CreateTablet for tablet 18156cbf5dc94bd2a4b98a1e521fc973 (DEFAULT_TABLE table=test-table [id=198bf811f6624bd5b967fd69ffb53c1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:35.538371 30601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18156cbf5dc94bd2a4b98a1e521fc973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:35.539140 30677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:35.527629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:42483 (local address 127.0.0.1:58270)
0929 03:18:35.528249 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:35.528264 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:35.528349 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:35.535195 (+ 6846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:35.535198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:35.535222 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:35.535324 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:35.535329 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.536652 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.536656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:35.537592 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:35.537604 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.538729 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.538733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:35.538824 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:35.538850 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 03:18:35.538931 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 03:18:35.539005 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":79,"threads_started":1}
I20260929 03:18:35.539196 30678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:35.527929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58270 (local address 127.13.238.130:42483)
0929 03:18:35.534647 (+ 6718us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:35.534658 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:35.534685 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:18:35.534780 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:35.534786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:35.535020 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:35.535459 (+ 439us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:35.535639 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.536525 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.537764 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.538597 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.538738 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:35.538881 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:35.538965 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 03:18:35.539056 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6599,"thread_start_us":115,"threads_started":1}
I20260929 03:18:35.541868 30469 tablet_service.cc:1467] Processing CreateTablet for tablet 18156cbf5dc94bd2a4b98a1e521fc973 (DEFAULT_TABLE table=test-table [id=198bf811f6624bd5b967fd69ffb53c1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:18:35.542866 30469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18156cbf5dc94bd2a4b98a1e521fc973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:18:35.545434 30680 tablet_bootstrap.cc:492] T 18156cbf5dc94bd2a4b98a1e521fc973 P d9d64153acee49f7bc033648fb6afac7: Bootstrap starting.
I20260929 03:18:35.548657 30680 tablet_bootstrap.cc:654] T 18156cbf5dc94bd2a4b98a1e521fc973 P d9d64153acee49f7bc033648fb6afac7: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:35.549785 30680 log.cc:824] T 18156cbf5dc94bd2a4b98a1e521fc973 P d9d64153acee49f7bc033648fb6afac7: Log is configured to *not* fsync() on all Append() calls
I20260929 03:18:35.550266 30681 tablet_bootstrap.cc:492] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58: Bootstrap starting.
I20260929 03:18:35.552091 30680 tablet_bootstrap.cc:492] T 18156cbf5dc94bd2a4b98a1e521fc973 P d9d64153acee49f7bc033648fb6afac7: No bootstrap required, opened a new log
I20260929 03:18:35.552304 30680 ts_tablet_manager.cc:1397] T 18156cbf5dc94bd2a4b98a1e521fc973 P d9d64153acee49f7bc033648fb6afac7: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 03:18:35.552884 30681 tablet_bootstrap.cc:654] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58: Neither blocks nor log segments found. Creating new log.
I20260929 03:18:35.553706 30681 log.cc:824] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58: Log is configured to *not* fsync() on all Append() calls
I20260929 03:18:35.555847 30681 tablet_bootstrap.cc:492] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58: No bootstrap required, opened a new log
I20260929 03:18:35.556035 30681 ts_tablet_manager.cc:1397] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260929 03:18:35.556679 30680 raft_consensus.cc:348] T 18156cbf5dc94bd2a4b98a1e521fc973 P d9d64153acee49f7bc033648fb6afac7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
I20260929 03:18:35.557024 30680 raft_consensus.cc:374] T 18156cbf5dc94bd2a4b98a1e521fc973 P d9d64153acee49f7bc033648fb6afac7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:35.557085 30680 raft_consensus.cc:729] T 18156cbf5dc94bd2a4b98a1e521fc973 P d9d64153acee49f7bc033648fb6afac7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d64153acee49f7bc033648fb6afac7, State: Initialized, Role: FOLLOWER
I20260929 03:18:35.557580 30680 consensus_queue.cc:260] T 18156cbf5dc94bd2a4b98a1e521fc973 P d9d64153acee49f7bc033648fb6afac7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
I20260929 03:18:35.558333 30667 heartbeater.cc:503] Master 127.13.238.190:42831 was elected leader, sending a full tablet report...
I20260929 03:18:35.558691 30680 ts_tablet_manager.cc:1428] T 18156cbf5dc94bd2a4b98a1e521fc973 P d9d64153acee49f7bc033648fb6afac7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 03:18:35.559367 30681 raft_consensus.cc:348] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
I20260929 03:18:35.559628 30681 raft_consensus.cc:374] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:18:35.559681 30681 raft_consensus.cc:729] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a75ce578b5e4782980fdb5857faaf58, State: Initialized, Role: FOLLOWER
I20260929 03:18:35.560314 30681 consensus_queue.cc:260] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
W20260929 03:18:35.560523 30398 tablet.cc:2369] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:18:35.561349 30681 ts_tablet_manager.cc:1428] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260929 03:18:35.561679 30535 heartbeater.cc:503] Master 127.13.238.190:42831 was elected leader, sending a full tablet report...
I20260929 03:18:35.619166 30685 raft_consensus.cc:482] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:18:35.619386 30685 raft_consensus.cc:504] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
I20260929 03:18:35.620612 30685 leader_election.cc:288] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9d64153acee49f7bc033648fb6afac7 (127.13.238.131:41419), eb3a7ddf42d54d05a6638a441d6c48e0 (127.13.238.129:43999)
I20260929 03:18:35.624356 30537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:35.620821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:41419 (local address 127.13.238.130:39491)
0929 03:18:35.621025 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:35.621041 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:35.621141 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:35.621412 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:35.621415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:35.621435 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:35.621562 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:35.621569 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.622397 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.622400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:35.623253 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:35.623263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.623943 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.623946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:35.624065 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:35.624083 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 03:18:35.624150 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 03:18:35.624217 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260929 03:18:35.624403 30676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:35.620922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:39491 (local address 127.13.238.131:41419)
0929 03:18:35.621042 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:35.621046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:35.621180 (+ 134us) server_negotiation.cc:407] Connection header received
0929 03:18:35.621261 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:35.621265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:35.621320 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:35.621391 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:35.621660 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.622301 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.623412 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.623843 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.623943 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:35.623998 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:35.624200 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 03:18:35.624280 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 03:18:35.624519 30686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:35.620875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:43999 (local address 127.13.238.130:59359)
0929 03:18:35.621233 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:35.621246 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:35.621341 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:35.621647 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:35.621649 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:35.621666 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:18:35.621799 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:35.621806 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.622594 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.622597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:35.623414 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:35.623423 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.624162 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.624165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:35.624254 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:35.624274 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 03:18:35.624327 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 03:18:35.624413 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":133,"threads_started":1}
I20260929 03:18:35.624750 30687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:35.620976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:59359 (local address 127.13.238.129:43999)
0929 03:18:35.621204 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:35.621209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:35.621323 (+ 114us) server_negotiation.cc:407] Connection header received
0929 03:18:35.621454 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:35.621457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:35.621522 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:35.621639 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:18:35.621900 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.622490 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.623542 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.624060 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.624162 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:35.624336 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:35.624372 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 03:18:35.624622 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":69,"threads_started":1}
I20260929 03:18:35.625195 30621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18156cbf5dc94bd2a4b98a1e521fc973" candidate_uuid: "2a75ce578b5e4782980fdb5857faaf58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d64153acee49f7bc033648fb6afac7" is_pre_election: true
I20260929 03:18:35.625190 30351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18156cbf5dc94bd2a4b98a1e521fc973" candidate_uuid: "2a75ce578b5e4782980fdb5857faaf58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" is_pre_election: true
I20260929 03:18:35.625480 30621 raft_consensus.cc:2454] T 18156cbf5dc94bd2a4b98a1e521fc973 P d9d64153acee49f7bc033648fb6afac7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a75ce578b5e4782980fdb5857faaf58 in term 0.
I20260929 03:18:35.625464 30351 raft_consensus.cc:2454] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a75ce578b5e4782980fdb5857faaf58 in term 0.
I20260929 03:18:35.625928 30425 leader_election.cc:302] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a75ce578b5e4782980fdb5857faaf58, d9d64153acee49f7bc033648fb6afac7; no voters:
I20260929 03:18:35.626180 30685 raft_consensus.cc:2781] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:18:35.626291 30685 raft_consensus.cc:482] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:18:35.626349 30685 raft_consensus.cc:3037] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:35.627384 30685 raft_consensus.cc:504] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
I20260929 03:18:35.627708 30685 leader_election.cc:288] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58 [CANDIDATE]: Term 1 election: Requested vote from peers d9d64153acee49f7bc033648fb6afac7 (127.13.238.131:41419), eb3a7ddf42d54d05a6638a441d6c48e0 (127.13.238.129:43999)
I20260929 03:18:35.628093 30621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18156cbf5dc94bd2a4b98a1e521fc973" candidate_uuid: "2a75ce578b5e4782980fdb5857faaf58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d64153acee49f7bc033648fb6afac7"
I20260929 03:18:35.628153 30351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18156cbf5dc94bd2a4b98a1e521fc973" candidate_uuid: "2a75ce578b5e4782980fdb5857faaf58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0"
I20260929 03:18:35.628204 30621 raft_consensus.cc:3037] T 18156cbf5dc94bd2a4b98a1e521fc973 P d9d64153acee49f7bc033648fb6afac7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:35.628278 30351 raft_consensus.cc:3037] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:18:35.629281 30351 raft_consensus.cc:2454] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a75ce578b5e4782980fdb5857faaf58 in term 1.
I20260929 03:18:35.629299 30621 raft_consensus.cc:2454] T 18156cbf5dc94bd2a4b98a1e521fc973 P d9d64153acee49f7bc033648fb6afac7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a75ce578b5e4782980fdb5857faaf58 in term 1.
I20260929 03:18:35.629568 30423 leader_election.cc:302] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a75ce578b5e4782980fdb5857faaf58, eb3a7ddf42d54d05a6638a441d6c48e0; no voters:
I20260929 03:18:35.629727 30685 raft_consensus.cc:2781] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:18:35.629928 30685 raft_consensus.cc:686] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58 [term 1 LEADER]: Becoming Leader. State: Replica: 2a75ce578b5e4782980fdb5857faaf58, State: Running, Role: LEADER
I20260929 03:18:35.630270 30685 consensus_queue.cc:237] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
I20260929 03:18:35.633621 30208 catalog_manager.cc:5495] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a75ce578b5e4782980fdb5857faaf58 (127.13.238.130). New cstate: current_term: 1 leader_uuid: "2a75ce578b5e4782980fdb5857faaf58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } health_report { overall_health: UNKNOWN } } }
I20260929 03:18:35.668125 30678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:35.664135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58286 (local address 127.13.238.130:42483)
0929 03:18:35.664261 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:35.664266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:35.664312 (+ 46us) server_negotiation.cc:407] Connection header received
0929 03:18:35.664534 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:35.664537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:35.664608 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:35.664704 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:18:35.665022 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.665540 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.666564 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:35.666787 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:35.666885 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:35.667196 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:18:35.667672 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:18:35.667767 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:35.667919 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 03:18:35.667979 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:18:35.674451 30621 raft_consensus.cc:1261] T 18156cbf5dc94bd2a4b98a1e521fc973 P d9d64153acee49f7bc033648fb6afac7 [term 1 FOLLOWER]: Refusing update from remote peer 2a75ce578b5e4782980fdb5857faaf58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:18:35.674445 30351 raft_consensus.cc:1261] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 FOLLOWER]: Refusing update from remote peer 2a75ce578b5e4782980fdb5857faaf58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:18:35.675136 30688 consensus_queue.cc:1035] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:18:35.675390 30685 consensus_queue.cc:1035] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:18:35.686632 30691 mvcc.cc:205] Tried to move back new op lower bound from 7334510246594682880 to 7334510246424117248. Current Snapshot: MvccSnapshot[applied={T|T < 7334510246594682880 or (T in {7334510246594682880})}]
I20260929 03:18:35.691145 30208 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:18:35.691365 30208 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:18:35.694057 30208 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce: Sending DeleteTablet for 3 replicas of tablet 18156cbf5dc94bd2a4b98a1e521fc973
I20260929 03:18:35.694844 30601 tablet_service.cc:1514] Processing DeleteTablet for tablet 18156cbf5dc94bd2a4b98a1e521fc973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:35 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:55904
I20260929 03:18:35.694860 30469 tablet_service.cc:1514] Processing DeleteTablet for tablet 18156cbf5dc94bd2a4b98a1e521fc973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:35 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:58270
I20260929 03:18:35.695073 30331 tablet_service.cc:1514] Processing DeleteTablet for tablet 18156cbf5dc94bd2a4b98a1e521fc973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:18:35 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:49122
I20260929 03:18:35.695422 30702 tablet_replica.cc:330] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0: stopping tablet replica
I20260929 03:18:35.695565 30702 raft_consensus.cc:2229] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:18:35.695601 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 30247
I20260929 03:18:35.696380 30700 tablet_replica.cc:330] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58: stopping tablet replica
I20260929 03:18:35.696755 30700 raft_consensus.cc:2229] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:18:35.697153 30700 raft_consensus.cc:2258] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:18:35.697520 30701 tablet_replica.cc:330] T 18156cbf5dc94bd2a4b98a1e521fc973 P d9d64153acee49f7bc033648fb6afac7: stopping tablet replica
I20260929 03:18:35.697825 30701 raft_consensus.cc:2229] T 18156cbf5dc94bd2a4b98a1e521fc973 P d9d64153acee49f7bc033648fb6afac7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:18:35.698182 30701 raft_consensus.cc:2258] T 18156cbf5dc94bd2a4b98a1e521fc973 P d9d64153acee49f7bc033648fb6afac7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:18:35.699118 30700 ts_tablet_manager.cc:1905] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:18:35.699837 30701 ts_tablet_manager.cc:1905] T 18156cbf5dc94bd2a4b98a1e521fc973 P d9d64153acee49f7bc033648fb6afac7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:18:35.702081 30700 ts_tablet_manager.cc:1918] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:18:35.702280 30700 log.cc:1196] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/wal/wals/18156cbf5dc94bd2a4b98a1e521fc973
I20260929 03:18:35.702414 30701 ts_tablet_manager.cc:1918] T 18156cbf5dc94bd2a4b98a1e521fc973 P d9d64153acee49f7bc033648fb6afac7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:18:35.702510 30701 log.cc:1196] T 18156cbf5dc94bd2a4b98a1e521fc973 P d9d64153acee49f7bc033648fb6afac7: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/18156cbf5dc94bd2a4b98a1e521fc973
W20260929 03:18:35.702785 30194 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.238.129:43999 (error 108)
I20260929 03:18:35.702975 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 30401
I20260929 03:18:35.703840 30701 ts_tablet_manager.cc:1939] T 18156cbf5dc94bd2a4b98a1e521fc973 P d9d64153acee49f7bc033648fb6afac7: Deleting consensus metadata
I20260929 03:18:35.705286 30677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:35.704977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:43999 (local address 127.0.0.1:49124)
0929 03:18:35.705113 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:35.705191 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.129:43999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 03:18:35.705272 30196 catalog_manager.cc:4841] TS d9d64153acee49f7bc033648fb6afac7 (127.13.238.131:41419): tablet 18156cbf5dc94bd2a4b98a1e521fc973 (table test-table [id=198bf811f6624bd5b967fd69ffb53c1c]) successfully deleted
W20260929 03:18:35.705494 30194 catalog_manager.cc:4584] TS eb3a7ddf42d54d05a6638a441d6c48e0 (127.13.238.129:43999): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 18156cbf5dc94bd2a4b98a1e521fc973: Network error: Client connection negotiation failed: client connection to 127.13.238.129:43999: connect: Connection refused (error 111)
I20260929 03:18:35.709476 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 30538
I20260929 03:18:35.709666 30677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:35.709386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:42483 (local address 127.0.0.1:58300)
0929 03:18:35.709524 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:35.709581 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.130:42483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 03:18:35.714915 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 30176
I20260929 03:18:55.720899 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:42831
--webserver_interface=127.13.238.190
--webserver_port=33565
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:42831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:18:55.833395 30704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:55.833676 30704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:55.833719 30704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:55.837776 30704 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 03:18:55.837828 30704 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 03:18:55.837842 30704 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 03:18:55.837862 30704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:18:55.837879 30704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:55.837896 30704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:55.837910 30704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:18:55.837927 30704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:18:55.843255 30704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:42831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:42831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=33565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:18:55.844345 30704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:55.845394 30704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:55.853947 30709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:55.854017 30712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:55.854127 30704 server_base.cc:1034] running on GCE node
W20260929 03:18:55.853955 30710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:55.854871 30704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:55.855959 30704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:55.857116 30704 hybrid_clock.cc:648] HybridClock initialized: now 1790651935857098 us; error 30 us; skew 500 ppm
Sep 29 03:18:55 dist-test-slave-nbjg krb5kdc[29627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651935, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:55.859884 30704 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:18:55.860795 30704 webserver.cc:460] Webserver started at http://127.13.238.190:33565/ using document root <none> and password file <none>
I20260929 03:18:55.861342 30704 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:55.861407 30704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:55.865299 30704 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:18:55.867568 30719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:55.868940 30704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:18:55.869091 30704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "6a674960548b41ef9bdd6508f2e421ce"
format_stamp: "Formatted at 2026-09-29 03:18:13 on dist-test-slave-nbjg"
server_key: "491f9449b5027cd553d1083c533c5607"
server_key_iv: "01ca876096e726f385b02b4b205811ce"
server_key_version: "encryptionkey@0"
I20260929 03:18:55.869632 30704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:55.902884 30704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:55.906181 30704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:55.906399 30704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:55.913432 30704 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:42831
I20260929 03:18:55.913686 30771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:42831 every 8 connection(s)
I20260929 03:18:55.914718 30704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:18:55.917820 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 30704
I20260929 03:18:55.918322 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:43999
--local_ip_for_outbound_sockets=127.13.238.129
--tserver_master_addrs=127.13.238.190:42831
--webserver_port=45617
--webserver_interface=127.13.238.129
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:18:55.920233 30772 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:18:55.921945 30772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce: Bootstrap starting.
I20260929 03:18:55.939096 30772 log.cc:824] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce: Log is configured to *not* fsync() on all Append() calls
I20260929 03:18:55.950073 30772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:18:55.950459 30772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce: Bootstrap complete.
I20260929 03:18:55.954382 30772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a674960548b41ef9bdd6508f2e421ce" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42831 } }
I20260929 03:18:55.954782 30772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a674960548b41ef9bdd6508f2e421ce, State: Initialized, Role: FOLLOWER
I20260929 03:18:55.955271 30772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a674960548b41ef9bdd6508f2e421ce" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42831 } }
I20260929 03:18:55.955390 30772 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:18:55.955451 30772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:18:55.955542 30772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [term 2 FOLLOWER]: Advancing to term 3
I20260929 03:18:55.957470 30772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a674960548b41ef9bdd6508f2e421ce" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42831 } }
I20260929 03:18:55.957793 30772 leader_election.cc:302] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a674960548b41ef9bdd6508f2e421ce; no voters:
I20260929 03:18:55.958092 30772 leader_election.cc:288] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 03:18:55.958221 30776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [term 3 FOLLOWER]: Leader election won for term 3
I20260929 03:18:55.958418 30776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [term 3 LEADER]: Becoming Leader. State: Replica: 6a674960548b41ef9bdd6508f2e421ce, State: Running, Role: LEADER
I20260929 03:18:55.958863 30776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a674960548b41ef9bdd6508f2e421ce" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42831 } }
I20260929 03:18:55.959465 30772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:18:55.959961 30776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a674960548b41ef9bdd6508f2e421ce. Latest consensus state: current_term: 3 leader_uuid: "6a674960548b41ef9bdd6508f2e421ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a674960548b41ef9bdd6508f2e421ce" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42831 } } }
I20260929 03:18:55.960160 30776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [sys.catalog]: This master's current role is: LEADER
I20260929 03:18:55.960309 30777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6a674960548b41ef9bdd6508f2e421ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a674960548b41ef9bdd6508f2e421ce" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42831 } } }
I20260929 03:18:55.960441 30777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce [sys.catalog]: This master's current role is: LEADER
I20260929 03:18:55.960664 30782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:18:55.966084 30782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:18:55.966845 30782 catalog_manager.cc:1259] Loaded cluster ID: 3dd2aa39bee746e5aea28c03060e1e39
I20260929 03:18:55.966914 30782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:18:55.969444 30782 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:18:55.970280 30782 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce: Loaded TSK: 0
I20260929 03:18:55.970369 30782 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6a674960548b41ef9bdd6508f2e421ce: Loaded TSK: 1
F20260929 03:18:55.971170 30782 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f21621919ad google::LogMessage::Fail() at ??:0
@ 0x7f216219488a google::LogMessage::SendToLog() at ??:0
@ 0x7f21621914de google::LogMessage::Flush() at ??:0
@ 0x7f2162192a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2164e5518d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f2164e4df0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f216caf67bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f216cad099c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f216cad1178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f216cb12df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f216cb3182b std::function<>::operator()() at ??:0
@ 0x7f216cad0b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f216cad1f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f216cace579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f216cb12640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f21643e33d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f21643e3c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f21643e53f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f21643d4512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2165a186db start_thread at ??:0
@ 0x7f2160c8f61f clone at ??:0
W20260929 03:18:56.062002 30774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:56.062441 30774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:56.062525 30774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:56.068658 30774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:56.068743 30774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:56.068835 30774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:18:56.077003 30774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:43999
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=45617
--tserver_master_addrs=127.13.238.190:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:18:56.078639 30774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:56.080240 30774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:56.092017 30819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:56.092010 30820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:56.092010 30822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:56.092293 30774 server_base.cc:1034] running on GCE node
I20260929 03:18:56.092875 30774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:56.093667 30774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:56.094873 30774 hybrid_clock.cc:648] HybridClock initialized: now 1790651936094838 us; error 51 us; skew 500 ppm
Sep 29 03:18:56 dist-test-slave-nbjg krb5kdc[29627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651936, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:56.098428 30774 init.cc:377] Logged in from keytab as kudu/127.13.238.129@KRBTEST.COM (short username kudu)
I20260929 03:18:56.099641 30774 webserver.cc:460] Webserver started at http://127.13.238.129:45617/ using document root <none> and password file <none>
I20260929 03:18:56.100417 30774 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:56.100513 30774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:56.105971 30774 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260929 03:18:56.108877 30829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:56.110597 30774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:18:56.110742 30774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "eb3a7ddf42d54d05a6638a441d6c48e0"
format_stamp: "Formatted at 2026-09-29 03:18:13 on dist-test-slave-nbjg"
server_key: "447755d56fbd7aa9d1da9953ed2addfd"
server_key_iv: "eef7261c1439168672bdbbb926395303"
server_key_version: "encryptionkey@0"
I20260929 03:18:56.111461 30774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:56.136054 30774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:56.140652 30774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:56.140893 30774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:56.141724 30774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:18:56.143595 30836 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 03:18:56.147996 30774 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 03:18:56.148092 30774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260929 03:18:56.148167 30774 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 03:18:56.150579 30774 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 03:18:56.150648 30774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260929 03:18:56.150975 30836 tablet_bootstrap.cc:492] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0: Bootstrap starting.
I20260929 03:18:56.162020 30774 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:43999
I20260929 03:18:56.162369 30943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:43999 every 8 connection(s)
I20260929 03:18:56.163399 30774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:18:56.165944 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 30774
I20260929 03:18:56.166499 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:42483
--local_ip_for_outbound_sockets=127.13.238.130
--tserver_master_addrs=127.13.238.190:42831
--webserver_port=37135
--webserver_interface=127.13.238.130
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:18:56.171234 30836 log.cc:824] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0: Log is configured to *not* fsync() on all Append() calls
I20260929 03:18:56.181321 30836 tablet_bootstrap.cc:492] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:18:56.181766 30836 tablet_bootstrap.cc:492] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0: Bootstrap complete.
I20260929 03:18:56.183103 30836 ts_tablet_manager.cc:1397] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.012s
I20260929 03:18:56.186393 30836 raft_consensus.cc:348] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
I20260929 03:18:56.186967 30836 raft_consensus.cc:729] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb3a7ddf42d54d05a6638a441d6c48e0, State: Initialized, Role: FOLLOWER
I20260929 03:18:56.187520 30836 consensus_queue.cc:260] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
I20260929 03:18:56.188611 30836 ts_tablet_manager.cc:1428] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
W20260929 03:18:56.219053 30946 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:18:56.166008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.129:41491)
0929 03:18:56.167365 (+ 1357us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:56.167429 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:56.168529 (+ 1100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:56.218160 (+ 49631us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1105,"thread_start_us":203,"threads_started":1}
W20260929 03:18:56.219566 30944 heartbeater.cc:644] Failed to heartbeat to 127.13.238.190:42831 (0 consecutive failures): Network error: Failed to ping master at 127.13.238.190:42831: Client connection negotiation failed: client connection to 127.13.238.190:42831: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 03:18:56.220377 30946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:56.220081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.129:44039)
0929 03:18:56.220238 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:56.220312 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 03:18:56.221037 30946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:56.220835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.129:36571)
0929 03:18:56.220941 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:56.220985 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 03:18:56.221187 30944 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:18:56.289671 30947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:56.290000 30947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:56.290066 30947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:56.294175 30947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:56.294271 30947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:56.294358 30947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:18:56.299854 30947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:42483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=37135
--tserver_master_addrs=127.13.238.190:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:18:56.300945 30947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:56.302018 30947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:56.311873 30954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:56.311866 30947 server_base.cc:1034] running on GCE node
W20260929 03:18:56.311865 30955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:56.311908 30957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:56.312537 30947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:56.313171 30947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:56.314328 30947 hybrid_clock.cc:648] HybridClock initialized: now 1790651936314302 us; error 32 us; skew 500 ppm
Sep 29 03:18:56 dist-test-slave-nbjg krb5kdc[29627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651936, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:56.317219 30947 init.cc:377] Logged in from keytab as kudu/127.13.238.130@KRBTEST.COM (short username kudu)
I20260929 03:18:56.318185 30947 webserver.cc:460] Webserver started at http://127.13.238.130:37135/ using document root <none> and password file <none>
I20260929 03:18:56.318768 30947 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:56.318850 30947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:56.322839 30947 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 03:18:56.325270 30964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:56.326615 30947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:18:56.326727 30947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "2a75ce578b5e4782980fdb5857faaf58"
format_stamp: "Formatted at 2026-09-29 03:18:14 on dist-test-slave-nbjg"
server_key: "c3fab53dc74dfa61b85710d05a7355e2"
server_key_iv: "77a035d42472b51565293e2b0fde7b1c"
server_key_version: "encryptionkey@0"
I20260929 03:18:56.327365 30947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:56.351859 30947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:56.355312 30947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:56.355504 30947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:56.356163 30947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:18:56.357386 30971 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260929 03:18:56.361402 30971 ts_tablet_manager.cc:1852] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 03:18:56.361799 30971 ts_tablet_manager.cc:1905] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:18:56.365872 30971 ts_tablet_manager.cc:1918] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:18:56.366006 30971 log.cc:1196] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/wal/wals/18156cbf5dc94bd2a4b98a1e521fc973
I20260929 03:18:56.366333 30971 ts_tablet_manager.cc:1939] T 18156cbf5dc94bd2a4b98a1e521fc973 P 2a75ce578b5e4782980fdb5857faaf58: Deleting consensus metadata
I20260929 03:18:56.366601 30947 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 03:18:56.366664 30947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260929 03:18:56.366796 30947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:18:56.366824 30947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:56.375581 30947 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:42483
I20260929 03:18:56.375648 31078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:42483 every 8 connection(s)
I20260929 03:18:56.376535 30947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:18:56.380283 31081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:56.378907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.130:51631)
0929 03:18:56.379422 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:56.379537 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":115,"threads_started":1}
W20260929 03:18:56.380697 31079 heartbeater.cc:644] Failed to heartbeat to 127.13.238.190:42831 (0 consecutive failures): Network error: Failed to ping master at 127.13.238.190:42831: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
I20260929 03:18:56.381274 31081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:56.381082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.130:59933)
0929 03:18:56.381176 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:56.381221 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 03:18:56.381817 31081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:56.381605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.130:58841)
0929 03:18:56.381715 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:56.381758 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 03:18:56.381953 31079 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 03:18:56.386835 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 30947
I20260929 03:18:56.387210 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:41419
--local_ip_for_outbound_sockets=127.13.238.131
--tserver_master_addrs=127.13.238.190:42831
--webserver_port=36987
--webserver_interface=127.13.238.131
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:18:56.499089 31082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:18:56.499363 31082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:18:56.499404 31082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:18:56.503407 31082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:18:56.503463 31082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:18:56.503520 31082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:18:56.508723 31082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:41419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=36987
--tserver_master_addrs=127.13.238.190:42831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:18:56.509744 31082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:18:56.510762 31082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:18:56.520290 31087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:56.520362 31090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:18:56.520339 31088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:18:56.520550 31082 server_base.cc:1034] running on GCE node
I20260929 03:18:56.520959 31082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:18:56.521585 31082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:18:56.522749 31082 hybrid_clock.cc:648] HybridClock initialized: now 1790651936522736 us; error 62 us; skew 500 ppm
Sep 29 03:18:56 dist-test-slave-nbjg krb5kdc[29627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651936, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:18:56.525424 31082 init.cc:377] Logged in from keytab as kudu/127.13.238.131@KRBTEST.COM (short username kudu)
I20260929 03:18:56.526316 31082 webserver.cc:460] Webserver started at http://127.13.238.131:36987/ using document root <none> and password file <none>
I20260929 03:18:56.526860 31082 fs_manager.cc:359] Metadata directory not provided
I20260929 03:18:56.526919 31082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:18:56.530853 31082 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 03:18:56.533001 31097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:56.534409 31082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:18:56.534516 31082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "d9d64153acee49f7bc033648fb6afac7"
format_stamp: "Formatted at 2026-09-29 03:18:14 on dist-test-slave-nbjg"
server_key: "054d5b7090832269ea0e72551efab345"
server_key_iv: "2929b683a1ecd0d0804bff2130a025b1"
server_key_version: "encryptionkey@0"
I20260929 03:18:56.535086 31082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:18:56.554083 31082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:18:56.557332 31082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:18:56.557515 31082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:18:56.558142 31082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:18:56.559037 31082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:18:56.559084 31082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:56.559127 31082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:18:56.559156 31082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:18:56.567633 31082 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:41419
I20260929 03:18:56.567659 31210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:41419 every 8 connection(s)
I20260929 03:18:56.568638 31082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:18:56.572429 31213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:56.570853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.131:34749)
0929 03:18:56.571573 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:56.571690 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":113,"threads_started":1}
W20260929 03:18:56.572999 31211 heartbeater.cc:644] Failed to heartbeat to 127.13.238.190:42831 (0 consecutive failures): Network error: Failed to ping master at 127.13.238.190:42831: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
I20260929 03:18:56.573590 31213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:56.573373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.131:59399)
0929 03:18:56.573483 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:56.573532 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 03:18:56.573941 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 31082
I20260929 03:18:56.574293 31213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:56.574031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.131:42357)
0929 03:18:56.574151 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:56.574219 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 03:18:56.574458 31211 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 03:18:57.222302 31215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:57.221769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.129:55657)
0929 03:18:57.222047 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:57.222131 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":75,"threads_started":1}
I20260929 03:18:57.382874 31216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:57.382484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.130:36599)
0929 03:18:57.382724 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:57.382781 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":77,"threads_started":1}
I20260929 03:18:57.575491 31217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:57.575057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.131:50683)
0929 03:18:57.575310 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:57.575377 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
I20260929 03:18:57.651199 31218 raft_consensus.cc:482] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:18:57.651427 31218 raft_consensus.cc:504] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
I20260929 03:18:57.652590 31218 leader_election.cc:288] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9d64153acee49f7bc033648fb6afac7 (127.13.238.131:41419), 2a75ce578b5e4782980fdb5857faaf58 (127.13.238.130:42483)
Sep 29 03:18:57 dist-test-slave-nbjg krb5kdc[29627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651936, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for kudu/127.13.238.130@KRBTEST.COM
Sep 29 03:18:57 dist-test-slave-nbjg krb5kdc[29627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651936, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for kudu/127.13.238.131@KRBTEST.COM
I20260929 03:18:57.665838 31215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:57.652848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:42483 (local address 127.13.238.129:44007)
0929 03:18:57.652978 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:57.652993 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:57.653109 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:57.655052 (+ 1943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:57.655057 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:57.655581 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:57.656036 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:57.656050 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:57.657325 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:57.657327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:57.658317 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:57.658327 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:57.658994 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:57.658998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:57.659210 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:57.659816 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:57.659833 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:57.661295 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:57.663290 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:57.663295 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:57.663309 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:57.663556 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:57.663882 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:57.663885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:57.663887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:57.664021 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:57.664812 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:57.664818 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:57.665105 (+ 287us) client_negotiation.cc:769] Sending connection context
0929 03:18:57.665272 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 03:18:57.665637 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":168,"spinlock_wait_cycles":9856}
I20260929 03:18:57.665850 31220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:57.653012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:44007 (local address 127.13.238.130:42483)
0929 03:18:57.653337 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:57.653355 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:57.653403 (+ 48us) server_negotiation.cc:407] Connection header received
0929 03:18:57.654303 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:57.654311 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:57.654675 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:57.654965 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:57.656197 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:57.657172 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:57.658424 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:57.658876 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:57.659017 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:57.661523 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:57.661528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:57.661540 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:57.661568 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:57.663170 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:57.663661 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:57.663666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:57.663670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:57.663793 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:57.664105 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:57.664108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:57.664109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:57.664627 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:57.664761 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:57.665418 (+ 657us) server_negotiation.cc:299] Negotiation successful
0929 03:18:57.665667 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":54,"threads_started":1}
I20260929 03:18:57.665868 31219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:57.652848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:41419 (local address 127.13.238.129:60745)
0929 03:18:57.653303 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:57.653316 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:18:57.653412 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:18:57.655051 (+ 1639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:18:57.655058 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:18:57.655583 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:18:57.656035 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:57.656050 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:57.657322 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:57.657324 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:57.658319 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:18:57.658330 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:18:57.659025 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:18:57.659029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:18:57.659208 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:57.659865 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:18:57.659877 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:18:57.661605 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:18:57.663527 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:57.663531 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:57.663533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:57.663715 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:57.664027 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:18:57.664031 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:18:57.664033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:18:57.664119 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:18:57.664823 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:18:57.664825 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:18:57.665105 (+ 280us) client_negotiation.cc:769] Sending connection context
0929 03:18:57.665272 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 03:18:57.665706 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":389,"spinlock_wait_cycles":44544,"thread_start_us":62,"threads_started":1}
I20260929 03:18:57.665874 31221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:57.653140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:60745 (local address 127.13.238.131:41419)
0929 03:18:57.653558 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0929 03:18:57.653578 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 03:18:57.653611 (+ 33us) server_negotiation.cc:407] Connection header received
0929 03:18:57.654387 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:18:57.654394 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:18:57.654702 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:18:57.654973 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:18:57.656208 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:57.657176 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:57.658441 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:18:57.658903 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:18:57.659065 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:18:57.661797 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:18:57.661802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:18:57.661813 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:18:57.661849 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:18:57.663421 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:57.663825 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:57.663830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:57.663834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:57.663940 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:18:57.664213 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:18:57.664216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:18:57.664218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:18:57.664627 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:18:57.664761 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 03:18:57.665413 (+ 652us) server_negotiation.cc:299] Negotiation successful
0929 03:18:57.665685 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":73,"threads_started":1}
W20260929 03:18:57.668481 30833 leader_election.cc:341] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9d64153acee49f7bc033648fb6afac7 (127.13.238.131:41419): Not found: Tablet not found: 18156cbf5dc94bd2a4b98a1e521fc973
W20260929 03:18:57.668645 30832 leader_election.cc:341] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a75ce578b5e4782980fdb5857faaf58 (127.13.238.130:42483): Not found: Tablet not found: 18156cbf5dc94bd2a4b98a1e521fc973
I20260929 03:18:57.668723 30832 leader_election.cc:302] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb3a7ddf42d54d05a6638a441d6c48e0; no voters: 2a75ce578b5e4782980fdb5857faaf58, d9d64153acee49f7bc033648fb6afac7
I20260929 03:18:57.668975 31218 raft_consensus.cc:2726] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:18:58.223667 31222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:58.223187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.129:60825)
0929 03:18:58.223493 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:58.223560 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":80,"threads_started":1}
I20260929 03:18:58.384146 31223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:58.383659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.130:58073)
0929 03:18:58.383951 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:58.384019 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":77,"threads_started":1}
I20260929 03:18:58.576606 31224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:58.576200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.131:42787)
0929 03:18:58.576452 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:58.576512 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
I20260929 03:18:59.224946 31225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:59.224435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.129:48277)
0929 03:18:59.224731 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:59.224797 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":75,"threads_started":1}
I20260929 03:18:59.265309 31226 raft_consensus.cc:482] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:18:59.265440 31226 raft_consensus.cc:504] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
I20260929 03:18:59.265914 31226 leader_election.cc:288] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9d64153acee49f7bc033648fb6afac7 (127.13.238.131:41419), 2a75ce578b5e4782980fdb5857faaf58 (127.13.238.130:42483)
W20260929 03:18:59.266790 30833 leader_election.cc:341] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9d64153acee49f7bc033648fb6afac7 (127.13.238.131:41419): Not found: Tablet not found: 18156cbf5dc94bd2a4b98a1e521fc973
W20260929 03:18:59.266913 30832 leader_election.cc:341] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a75ce578b5e4782980fdb5857faaf58 (127.13.238.130:42483): Not found: Tablet not found: 18156cbf5dc94bd2a4b98a1e521fc973
I20260929 03:18:59.267011 30832 leader_election.cc:302] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb3a7ddf42d54d05a6638a441d6c48e0; no voters: 2a75ce578b5e4782980fdb5857faaf58, d9d64153acee49f7bc033648fb6afac7
I20260929 03:18:59.267166 31226 raft_consensus.cc:2726] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:18:59.385324 31227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:59.384858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.130:53227)
0929 03:18:59.385151 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:59.385210 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":74,"threads_started":1}
I20260929 03:18:59.577915 31228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:18:59.577405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.131:51127)
0929 03:18:59.577688 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 03:18:59.577773 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":96,"threads_started":1}
I20260929 03:19:00.226315 31229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:00.225780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.129:41825)
0929 03:19:00.226105 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:00.226169 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":79,"threads_started":1}
I20260929 03:19:00.386657 31230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:00.386107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.130:47429)
0929 03:19:00.386469 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:00.386551 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":124,"threads_started":1}
I20260929 03:19:00.579102 31231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:00.578684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.131:34489)
0929 03:19:00.578948 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:00.579003 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":86,"threads_started":1}
I20260929 03:19:00.946154 31232 raft_consensus.cc:482] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:19:00.946321 31232 raft_consensus.cc:504] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
I20260929 03:19:00.946708 31232 leader_election.cc:288] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9d64153acee49f7bc033648fb6afac7 (127.13.238.131:41419), 2a75ce578b5e4782980fdb5857faaf58 (127.13.238.130:42483)
W20260929 03:19:00.947433 30832 leader_election.cc:341] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a75ce578b5e4782980fdb5857faaf58 (127.13.238.130:42483): Not found: Tablet not found: 18156cbf5dc94bd2a4b98a1e521fc973
W20260929 03:19:00.947536 30833 leader_election.cc:341] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9d64153acee49f7bc033648fb6afac7 (127.13.238.131:41419): Not found: Tablet not found: 18156cbf5dc94bd2a4b98a1e521fc973
I20260929 03:19:00.947638 30833 leader_election.cc:302] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb3a7ddf42d54d05a6638a441d6c48e0; no voters: 2a75ce578b5e4782980fdb5857faaf58, d9d64153acee49f7bc033648fb6afac7
I20260929 03:19:00.947832 31232 raft_consensus.cc:2726] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:19:01.227880 31233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:01.227374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.129:39845)
0929 03:19:01.227704 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:01.227769 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":90,"threads_started":1}
I20260929 03:19:01.387895 31234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:01.387457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.130:37219)
0929 03:19:01.387724 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:01.387789 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":72,"threads_started":1}
I20260929 03:19:01.580307 31235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:01.579901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.131:53311)
0929 03:19:01.580140 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:01.580206 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":68,"threads_started":1}
I20260929 03:19:02.229240 31236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:02.228752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.129:44759)
0929 03:19:02.229073 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:02.229142 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":110,"threads_started":1}
I20260929 03:19:02.389165 31237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:02.388684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.130:60655)
0929 03:19:02.388993 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:02.389058 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":87,"threads_started":1}
I20260929 03:19:02.581629 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:02.581117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.131:54831)
0929 03:19:02.581464 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:02.581530 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":117,"threads_started":1}
I20260929 03:19:03.230633 31239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:03.230096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.129:49411)
0929 03:19:03.230415 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:03.230520 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":137,"threads_started":1}
I20260929 03:19:03.390517 31240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:03.389995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.130:43077)
0929 03:19:03.390344 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:03.390413 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":88,"threads_started":1}
I20260929 03:19:03.437870 31241 raft_consensus.cc:482] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:19:03.438022 31241 raft_consensus.cc:504] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
I20260929 03:19:03.438482 31241 leader_election.cc:288] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9d64153acee49f7bc033648fb6afac7 (127.13.238.131:41419), 2a75ce578b5e4782980fdb5857faaf58 (127.13.238.130:42483)
W20260929 03:19:03.439349 30833 leader_election.cc:341] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9d64153acee49f7bc033648fb6afac7 (127.13.238.131:41419): Not found: Tablet not found: 18156cbf5dc94bd2a4b98a1e521fc973
W20260929 03:19:03.439479 30832 leader_election.cc:341] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a75ce578b5e4782980fdb5857faaf58 (127.13.238.130:42483): Not found: Tablet not found: 18156cbf5dc94bd2a4b98a1e521fc973
I20260929 03:19:03.439546 30832 leader_election.cc:302] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb3a7ddf42d54d05a6638a441d6c48e0; no voters: 2a75ce578b5e4782980fdb5857faaf58, d9d64153acee49f7bc033648fb6afac7
I20260929 03:19:03.439698 31241 raft_consensus.cc:2726] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:19:03.582993 31242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:03.582454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.131:33079)
0929 03:19:03.582820 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:03.582889 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":92,"threads_started":1}
I20260929 03:19:04.231959 31243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:04.231500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.129:35023)
0929 03:19:04.231784 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:04.231853 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":75,"threads_started":1}
I20260929 03:19:04.391696 31244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:04.391265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.130:53903)
0929 03:19:04.391528 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:04.391589 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":79,"threads_started":1}
I20260929 03:19:04.584318 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:04.583851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.131:56629)
0929 03:19:04.584138 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:04.584202 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":113,"threads_started":1}
I20260929 03:19:05.233187 31246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:05.232713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.129:54697)
0929 03:19:05.233016 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:05.233076 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":97,"threads_started":1}
I20260929 03:19:05.393021 31247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:05.392558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.130:39395)
0929 03:19:05.392847 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:05.392916 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":79,"threads_started":1}
I20260929 03:19:05.585424 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:05.584959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.131:43433)
0929 03:19:05.585227 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:05.585318 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
I20260929 03:19:06.234589 31249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:06.234113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.129:60987)
0929 03:19:06.234406 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:06.234471 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":107,"threads_started":1}
I20260929 03:19:06.394366 31250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:06.393893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.130:60515)
0929 03:19:06.394152 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:06.394250 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":76,"threads_started":1}
I20260929 03:19:06.586771 31251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:06.586261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.131:35811)
0929 03:19:06.586575 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:06.586646 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":88,"threads_started":1}
I20260929 03:19:07.235958 31252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:07.235469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.129:42837)
0929 03:19:07.235781 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:07.235850 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":97,"threads_started":1}
I20260929 03:19:07.395584 31253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:07.395139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.130:38727)
0929 03:19:07.395416 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:07.395479 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":73,"threads_started":1}
I20260929 03:19:07.588126 31254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:07.587624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.131:48783)
0929 03:19:07.587928 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:07.588012 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":91,"threads_started":1}
I20260929 03:19:08.237355 31255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:08.236844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.129:35283)
0929 03:19:08.237183 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:08.237252 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":100,"threads_started":1}
I20260929 03:19:08.396827 31256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:08.396350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.130:36357)
0929 03:19:08.396645 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:08.396708 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":98,"threads_started":1}
I20260929 03:19:08.589455 31257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:08.588986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.131:59355)
0929 03:19:08.589282 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:08.589351 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":91,"threads_started":1}
I20260929 03:19:09.238737 31258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:09.238264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.129:47325)
0929 03:19:09.238562 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:09.238630 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":83,"threads_started":1}
I20260929 03:19:09.397984 31259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:09.397549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.130:50971)
0929 03:19:09.397816 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:09.397872 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":73,"threads_started":1}
I20260929 03:19:09.590855 31260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:09.590385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.131:52923)
0929 03:19:09.590681 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:09.590754 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":91,"threads_started":1}
I20260929 03:19:10.240025 31261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:10.239500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.129:55517)
0929 03:19:10.239841 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:10.239906 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":77,"threads_started":1}
I20260929 03:19:10.399315 31262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:10.398844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.130:39457)
0929 03:19:10.399141 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:10.399210 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":78,"threads_started":1}
I20260929 03:19:10.592133 31263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:10.591675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.131:36345)
0929 03:19:10.591963 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:10.592028 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":93,"threads_started":1}
I20260929 03:19:10.888790 31264 raft_consensus.cc:482] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:19:10.888932 31264 raft_consensus.cc:504] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d64153acee49f7bc033648fb6afac7" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 41419 } } peers { permanent_uuid: "2a75ce578b5e4782980fdb5857faaf58" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 42483 } } peers { permanent_uuid: "eb3a7ddf42d54d05a6638a441d6c48e0" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 43999 } }
I20260929 03:19:10.889312 31264 leader_election.cc:288] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9d64153acee49f7bc033648fb6afac7 (127.13.238.131:41419), 2a75ce578b5e4782980fdb5857faaf58 (127.13.238.130:42483)
W20260929 03:19:10.890226 30833 leader_election.cc:341] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d9d64153acee49f7bc033648fb6afac7 (127.13.238.131:41419): Not found: Tablet not found: 18156cbf5dc94bd2a4b98a1e521fc973
W20260929 03:19:10.890345 30832 leader_election.cc:341] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a75ce578b5e4782980fdb5857faaf58 (127.13.238.130:42483): Not found: Tablet not found: 18156cbf5dc94bd2a4b98a1e521fc973
I20260929 03:19:10.890410 30832 leader_election.cc:302] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb3a7ddf42d54d05a6638a441d6c48e0; no voters: 2a75ce578b5e4782980fdb5857faaf58, d9d64153acee49f7bc033648fb6afac7
I20260929 03:19:10.890552 31264 raft_consensus.cc:2726] T 18156cbf5dc94bd2a4b98a1e521fc973 P eb3a7ddf42d54d05a6638a441d6c48e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:19:11.241386 31265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:11.240962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.129:57561)
0929 03:19:11.241229 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:11.241285 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":69,"threads_started":1}
I20260929 03:19:11.400722 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:11.400251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42831 (local address 127.13.238.130:36511)
0929 03:19:11.400552 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:11.400620 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.190:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":86,"threads_started":1}
I20260929 03:19:11.582077 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 30774
I20260929 03:19:11.587770 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 30947
I20260929 03:19:11.592731 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 31082
2026-09-29T03:19:11Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59114 ms)
[----------] 30 tests from SecurityITest (238053 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:19:11 dist-test-slave-nbjg krb5kdc[31269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:19:11 dist-test-slave-nbjg krb5kdc[31269](info): set up 2 sockets
Sep 29 03:19:11 dist-test-slave-nbjg krb5kdc[31269](info): commencing operation
krb5kdc: starting...
W20260929 03:19:12.558250 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:19:12 dist-test-slave-nbjg krb5kdc[31269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:19:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:19:12.679651 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:43575
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:34501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:43575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:12.794234 31282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:12.794554 31282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:12.794615 31282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:12.798873 31282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:12.798966 31282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:12.798995 31282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:12.799017 31282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:12.799038 31282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:12.804714 31282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:43575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:43575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:12.806308 31282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:12.807541 31282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:12.817487 31290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:12.817538 31287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:12.817494 31288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:12.818081 31282 server_base.cc:1034] running on GCE node
I20260929 03:19:12.818742 31282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:12.820015 31282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:12.821210 31282 hybrid_clock.cc:648] HybridClock initialized: now 1790651952821187 us; error 36 us; skew 500 ppm
Sep 29 03:19:12 dist-test-slave-nbjg krb5kdc[31269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651952, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:12.824120 31282 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:19:12.825073 31282 webserver.cc:460] Webserver started at http://127.13.238.190:35461/ using document root <none> and password file <none>
I20260929 03:19:12.825641 31282 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:12.825723 31282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:12.825922 31282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:12.828055 31282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "717e4cc940b54761995f76f0e55fe6cc"
format_stamp: "Formatted at 2026-09-29 03:19:12 on dist-test-slave-nbjg"
server_key: "fe2cf02e2a1a6b4ac44eb3afdad62c6e"
server_key_iv: "99e5ce59ebac498b9aab16d3e824f210"
server_key_version: "encryptionkey@0"
I20260929 03:19:12.828585 31282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "717e4cc940b54761995f76f0e55fe6cc"
format_stamp: "Formatted at 2026-09-29 03:19:12 on dist-test-slave-nbjg"
server_key: "fe2cf02e2a1a6b4ac44eb3afdad62c6e"
server_key_iv: "99e5ce59ebac498b9aab16d3e824f210"
server_key_version: "encryptionkey@0"
I20260929 03:19:12.832415 31282 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:19:12.835111 31297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:12.836531 31282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:19:12.836647 31282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "717e4cc940b54761995f76f0e55fe6cc"
format_stamp: "Formatted at 2026-09-29 03:19:12 on dist-test-slave-nbjg"
server_key: "fe2cf02e2a1a6b4ac44eb3afdad62c6e"
server_key_iv: "99e5ce59ebac498b9aab16d3e824f210"
server_key_version: "encryptionkey@0"
I20260929 03:19:12.836750 31282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:12.863255 31282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:12.866637 31282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:12.866801 31282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:12.873660 31282 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:43575
I20260929 03:19:12.873687 31349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:43575 every 8 connection(s)
I20260929 03:19:12.874650 31282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:12.876606 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 31282
I20260929 03:19:12.876716 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:19:12.877012 14266 external_mini_cluster.cc:1428] Setting key d406da0400304160ee649985f0fc0644
I20260929 03:19:12.877696 31350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:12.883708 31350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 717e4cc940b54761995f76f0e55fe6cc: Bootstrap starting.
Sep 29 03:19:12 dist-test-slave-nbjg krb5kdc[31269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:19:12.886073 31350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 717e4cc940b54761995f76f0e55fe6cc: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:12.886937 31350 log.cc:824] T 00000000000000000000000000000000 P 717e4cc940b54761995f76f0e55fe6cc: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:12.888901 31350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 717e4cc940b54761995f76f0e55fe6cc: No bootstrap required, opened a new log
I20260929 03:19:12.890715 31353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:12.878486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41760 (local address 127.13.238.190:43575)
0929 03:19:12.878938 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:12.878954 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:12.878993 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:19:12.879693 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:12.879710 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:12.880054 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:12.880365 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:12.881160 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.882148 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.883009 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:12.883478 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:12.883628 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:12.885977 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:12.885982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:12.885996 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:12.886062 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:12.887924 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:12.888331 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:12.888336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:12.888343 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:12.888457 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:12.888706 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:12.888709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:12.888711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:12.889154 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:12.889322 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:12.889620 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 03:19:12.889912 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":154,"threads_started":1}
I20260929 03:19:12.892946 31350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 717e4cc940b54761995f76f0e55fe6cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717e4cc940b54761995f76f0e55fe6cc" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43575 } }
I20260929 03:19:12.893234 31350 raft_consensus.cc:374] T 00000000000000000000000000000000 P 717e4cc940b54761995f76f0e55fe6cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:12.893302 31350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 717e4cc940b54761995f76f0e55fe6cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717e4cc940b54761995f76f0e55fe6cc, State: Initialized, Role: FOLLOWER
I20260929 03:19:12.893723 31350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 717e4cc940b54761995f76f0e55fe6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717e4cc940b54761995f76f0e55fe6cc" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43575 } }
I20260929 03:19:12.893848 31350 raft_consensus.cc:388] T 00000000000000000000000000000000 P 717e4cc940b54761995f76f0e55fe6cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:12.893894 31350 raft_consensus.cc:482] T 00000000000000000000000000000000 P 717e4cc940b54761995f76f0e55fe6cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:12.893965 31350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 717e4cc940b54761995f76f0e55fe6cc [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:12.894905 31350 raft_consensus.cc:504] T 00000000000000000000000000000000 P 717e4cc940b54761995f76f0e55fe6cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717e4cc940b54761995f76f0e55fe6cc" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43575 } }
I20260929 03:19:12.895213 31350 leader_election.cc:302] T 00000000000000000000000000000000 P 717e4cc940b54761995f76f0e55fe6cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 717e4cc940b54761995f76f0e55fe6cc; no voters:
I20260929 03:19:12.895541 31350 leader_election.cc:288] T 00000000000000000000000000000000 P 717e4cc940b54761995f76f0e55fe6cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:19:12.895649 31355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 717e4cc940b54761995f76f0e55fe6cc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:12.895869 31355 raft_consensus.cc:686] T 00000000000000000000000000000000 P 717e4cc940b54761995f76f0e55fe6cc [term 1 LEADER]: Becoming Leader. State: Replica: 717e4cc940b54761995f76f0e55fe6cc, State: Running, Role: LEADER
I20260929 03:19:12.896209 31355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 717e4cc940b54761995f76f0e55fe6cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717e4cc940b54761995f76f0e55fe6cc" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43575 } }
I20260929 03:19:12.896873 31350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 717e4cc940b54761995f76f0e55fe6cc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:12.898059 31357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 717e4cc940b54761995f76f0e55fe6cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 717e4cc940b54761995f76f0e55fe6cc. Latest consensus state: current_term: 1 leader_uuid: "717e4cc940b54761995f76f0e55fe6cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717e4cc940b54761995f76f0e55fe6cc" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43575 } } }
I20260929 03:19:12.898231 31357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 717e4cc940b54761995f76f0e55fe6cc [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:12.898574 31356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 717e4cc940b54761995f76f0e55fe6cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "717e4cc940b54761995f76f0e55fe6cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717e4cc940b54761995f76f0e55fe6cc" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 43575 } } }
I20260929 03:19:12.898636 31364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:12.898710 31356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 717e4cc940b54761995f76f0e55fe6cc [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:12.901538 31364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:12.906942 31364 catalog_manager.cc:1347] Generated new cluster ID: 137be3f15a1b42828de1bec96744632e
I20260929 03:19:12.907011 31364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:12.922096 31364 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:12.923300 31364 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:12.928791 31364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 717e4cc940b54761995f76f0e55fe6cc: Generated new TSK 0
I20260929 03:19:12.929482 31364 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
I20260929 03:19:12.961457 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:0
--local_ip_for_outbound_sockets=127.13.238.129
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43575
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:34501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:13.106751 31376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:13.107053 31376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:13.107100 31376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:13.111282 31376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:13.111361 31376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:13.111423 31376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:19:13.117074 31376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:13.118372 31376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:13.119486 31376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:13.129272 31384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:13.129339 31381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:13.129508 31376 server_base.cc:1034] running on GCE node
W20260929 03:19:13.129364 31382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:13.130023 31376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:13.130723 31376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:13.131927 31376 hybrid_clock.cc:648] HybridClock initialized: now 1790651953131901 us; error 43 us; skew 500 ppm
Sep 29 03:19:13 dist-test-slave-nbjg krb5kdc[31269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651953, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:13.134670 31376 init.cc:377] Logged in from keytab as kudu/127.13.238.129@KRBTEST.COM (short username kudu)
I20260929 03:19:13.135528 31376 webserver.cc:460] Webserver started at http://127.13.238.129:43519/ using document root <none> and password file <none>
I20260929 03:19:13.136082 31376 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:13.136149 31376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:13.136348 31376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:13.138747 31376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-0/data/instance:
uuid: "63e71c018e9f4667a3fdcd5d38062989"
format_stamp: "Formatted at 2026-09-29 03:19:13 on dist-test-slave-nbjg"
server_key: "1513ccbf6b653adad2390cb5bf8b17ee"
server_key_iv: "e862aae367dc785eebb91a2159496a84"
server_key_version: "encryptionkey@0"
I20260929 03:19:13.139299 31376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance:
uuid: "63e71c018e9f4667a3fdcd5d38062989"
format_stamp: "Formatted at 2026-09-29 03:19:13 on dist-test-slave-nbjg"
server_key: "1513ccbf6b653adad2390cb5bf8b17ee"
server_key_iv: "e862aae367dc785eebb91a2159496a84"
server_key_version: "encryptionkey@0"
I20260929 03:19:13.143066 31376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:19:13.145423 31391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:13.146783 31376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:19:13.146890 31376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "63e71c018e9f4667a3fdcd5d38062989"
format_stamp: "Formatted at 2026-09-29 03:19:13 on dist-test-slave-nbjg"
server_key: "1513ccbf6b653adad2390cb5bf8b17ee"
server_key_iv: "e862aae367dc785eebb91a2159496a84"
server_key_version: "encryptionkey@0"
I20260929 03:19:13.146981 31376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:13.165892 31376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:13.169253 31376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:13.169441 31376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:13.170077 31376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:13.170974 31376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:13.171032 31376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:13.171090 31376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:13.171116 31376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:13.180065 31376 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:39671
I20260929 03:19:13.180087 31504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:39671 every 8 connection(s)
I20260929 03:19:13.180962 31376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:19:13.189154 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 31376
I20260929 03:19:13.189288 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance
I20260929 03:19:13.189605 14266 external_mini_cluster.cc:1428] Setting key 3f39e695414f10f0f813269f95a13dc4
Sep 29 03:19:13 dist-test-slave-nbjg krb5kdc[31269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651953, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:19:13.194960 31353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:13.182781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:50523 (local address 127.13.238.190:43575)
0929 03:19:13.182915 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:13.182919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:13.183799 (+ 880us) server_negotiation.cc:407] Connection header received
0929 03:19:13.185096 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:13.185099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:13.185156 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:13.185230 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:13.186467 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.187011 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.188149 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.188387 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.188479 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:13.191201 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:13.191205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:13.191208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:13.191235 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:13.192623 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:13.193157 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:13.193161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:13.193163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:13.193256 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:13.193629 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:13.193632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:13.193633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:13.193868 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:13.193964 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:13.194588 (+ 624us) server_negotiation.cc:299] Negotiation successful
0929 03:19:13.194734 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:19:13.195776 31507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:13.183179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43575 (local address 127.13.238.129:50523)
0929 03:19:13.183648 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:13.183692 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:13.184882 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:13.185424 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:13.185431 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:13.185790 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:13.186326 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:13.186343 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.187131 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.187134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:13.188031 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:13.188041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.188531 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.188534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:13.188697 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:13.189262 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:13.189284 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:13.191046 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:13.192754 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:13.192762 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:13.192781 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:13.193065 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:13.193364 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:13.193368 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:13.193369 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:13.193534 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:13.193998 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:13.194004 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:13.194334 (+ 330us) client_negotiation.cc:769] Sending connection context
0929 03:19:13.194525 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 03:19:13.194883 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":110,"threads_started":1}
I20260929 03:19:13.197276 31505 heartbeater.cc:347] Connected to a master server at 127.13.238.190:43575
I20260929 03:19:13.197575 31505 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:13.198103 31505 heartbeater.cc:511] Master 127.13.238.190:43575 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.238.130@KRBTEST.COM; defaulting to no policy
I20260929 03:19:13.199796 31312 ts_manager.cc:194] Registered new tserver with Master: 63e71c018e9f4667a3fdcd5d38062989 (127.13.238.129:39671)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.130@KRBTEST.COM" created.
I20260929 03:19:13.201346 31312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.129@KRBTEST.COM'} at 127.13.238.129:50523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
I20260929 03:19:13.218156 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:0
--local_ip_for_outbound_sockets=127.13.238.130
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43575
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:34501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:13.328869 31510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:13.329124 31510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:13.329165 31510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:13.333108 31510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:13.333168 31510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:13.333245 31510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:19:13.338562 31510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:13.339576 31510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:13.340644 31510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:13.350615 31516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:13.350612 31515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:13.350660 31518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:13.350879 31510 server_base.cc:1034] running on GCE node
I20260929 03:19:13.351476 31510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:13.352138 31510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:13.353327 31510 hybrid_clock.cc:648] HybridClock initialized: now 1790651953353304 us; error 36 us; skew 500 ppm
Sep 29 03:19:13 dist-test-slave-nbjg krb5kdc[31269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651953, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:13.356020 31510 init.cc:377] Logged in from keytab as kudu/127.13.238.130@KRBTEST.COM (short username kudu)
I20260929 03:19:13.356928 31510 webserver.cc:460] Webserver started at http://127.13.238.130:41153/ using document root <none> and password file <none>
I20260929 03:19:13.357483 31510 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:13.357547 31510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:13.357748 31510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:13.359828 31510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-1/data/instance:
uuid: "b5a278eecb104d6fa6df44f49269fe41"
format_stamp: "Formatted at 2026-09-29 03:19:13 on dist-test-slave-nbjg"
server_key: "8477b25e566341a14f424201b92c73a7"
server_key_iv: "95c8cea66bc19a7a89628f5628daaee6"
server_key_version: "encryptionkey@0"
I20260929 03:19:13.360369 31510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance:
uuid: "b5a278eecb104d6fa6df44f49269fe41"
format_stamp: "Formatted at 2026-09-29 03:19:13 on dist-test-slave-nbjg"
server_key: "8477b25e566341a14f424201b92c73a7"
server_key_iv: "95c8cea66bc19a7a89628f5628daaee6"
server_key_version: "encryptionkey@0"
I20260929 03:19:13.364082 31510 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 03:19:13.366504 31525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:13.367786 31510 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 03:19:13.367899 31510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "b5a278eecb104d6fa6df44f49269fe41"
format_stamp: "Formatted at 2026-09-29 03:19:13 on dist-test-slave-nbjg"
server_key: "8477b25e566341a14f424201b92c73a7"
server_key_iv: "95c8cea66bc19a7a89628f5628daaee6"
server_key_version: "encryptionkey@0"
I20260929 03:19:13.368031 31510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:13.395009 31510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:13.398162 31510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:13.398350 31510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:13.398916 31510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:13.399817 31510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:13.399885 31510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:13.399940 31510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:13.399973 31510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:13.408150 31510 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:35779
I20260929 03:19:13.408183 31638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:35779 every 8 connection(s)
I20260929 03:19:13.409009 31510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:19:13.414597 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 31510
I20260929 03:19:13.414714 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance
I20260929 03:19:13.415017 14266 external_mini_cluster.cc:1428] Setting key ae5d98747c496b8b6568682b9306598d
Sep 29 03:19:13 dist-test-slave-nbjg krb5kdc[31269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651953, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:19:13.423522 31353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:13.410769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:55907 (local address 127.13.238.190:43575)
0929 03:19:13.410934 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:13.410939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:13.411827 (+ 888us) server_negotiation.cc:407] Connection header received
0929 03:19:13.413128 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:13.413131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:13.413179 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:13.413248 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:13.414600 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.415373 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.416729 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.416956 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.417040 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:13.419609 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:13.419613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:13.419616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:13.419644 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:13.421109 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:13.421728 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:13.421731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:13.421733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:13.421813 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:13.422304 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:13.422308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:13.422309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:13.422520 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:13.422602 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:13.423271 (+ 669us) server_negotiation.cc:299] Negotiation successful
0929 03:19:13.423382 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.13.238.131@KRBTEST.COM; defaulting to no policy
I20260929 03:19:13.424701 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:13.411180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43575 (local address 127.13.238.130:55907)
0929 03:19:13.411674 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:13.411718 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:13.412926 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:13.413440 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:13.413447 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:13.413803 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:13.414445 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:13.414463 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.415516 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.415519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:13.416611 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:13.416624 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.417105 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.417109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:13.417323 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:13.417925 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:13.417945 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:13.419434 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:13.421230 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:13.421238 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:13.421257 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:13.421615 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:13.421952 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:13.421956 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:13.421958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:13.422155 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:13.422623 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:13.422631 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:13.422975 (+ 344us) client_negotiation.cc:769] Sending connection context
0929 03:19:13.423198 (+ 223us) client_negotiation.cc:240] Negotiation successful
0929 03:19:13.423555 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":114,"threads_started":1}
I20260929 03:19:13.426107 31639 heartbeater.cc:347] Connected to a master server at 127.13.238.190:43575
I20260929 03:19:13.426363 31639 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:13.426905 31639 heartbeater.cc:511] Master 127.13.238.190:43575 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.131@KRBTEST.COM" created.
I20260929 03:19:13.427897 31312 ts_manager.cc:194] Registered new tserver with Master: b5a278eecb104d6fa6df44f49269fe41 (127.13.238.130:35779)
I20260929 03:19:13.428506 31312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.130@KRBTEST.COM'} at 127.13.238.130:55907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
I20260929 03:19:13.445111 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:0
--local_ip_for_outbound_sockets=127.13.238.131
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43575
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:34501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:13.556798 31644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:13.557120 31644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:13.557184 31644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:13.561357 31644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:13.561419 31644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:13.561479 31644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:19:13.566938 31644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:43575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:13.567973 31644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:13.569015 31644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:13.578886 31649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:13.578958 31644 server_base.cc:1034] running on GCE node
W20260929 03:19:13.578907 31650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:13.578881 31652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:13.579594 31644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:13.580245 31644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:13.581401 31644 hybrid_clock.cc:648] HybridClock initialized: now 1790651953581375 us; error 34 us; skew 500 ppm
Sep 29 03:19:13 dist-test-slave-nbjg krb5kdc[31269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651953, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:13.584295 31644 init.cc:377] Logged in from keytab as kudu/127.13.238.131@KRBTEST.COM (short username kudu)
I20260929 03:19:13.585306 31644 webserver.cc:460] Webserver started at http://127.13.238.131:35443/ using document root <none> and password file <none>
I20260929 03:19:13.585901 31644 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:13.585970 31644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:13.586179 31644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:13.588450 31644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-2/data/instance:
uuid: "a09fc001b595468589711e080703abaf"
format_stamp: "Formatted at 2026-09-29 03:19:13 on dist-test-slave-nbjg"
server_key: "4b9e018f00e35eccfdb595d7045860fd"
server_key_iv: "2d856b50fe7041319a501964dbcdbe63"
server_key_version: "encryptionkey@0"
I20260929 03:19:13.589028 31644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance:
uuid: "a09fc001b595468589711e080703abaf"
format_stamp: "Formatted at 2026-09-29 03:19:13 on dist-test-slave-nbjg"
server_key: "4b9e018f00e35eccfdb595d7045860fd"
server_key_iv: "2d856b50fe7041319a501964dbcdbe63"
server_key_version: "encryptionkey@0"
I20260929 03:19:13.593299 31644 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:19:13.596074 31659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:13.597570 31644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:19:13.597697 31644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "a09fc001b595468589711e080703abaf"
format_stamp: "Formatted at 2026-09-29 03:19:13 on dist-test-slave-nbjg"
server_key: "4b9e018f00e35eccfdb595d7045860fd"
server_key_iv: "2d856b50fe7041319a501964dbcdbe63"
server_key_version: "encryptionkey@0"
I20260929 03:19:13.597806 31644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:13.615211 31644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:13.618445 31644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:13.618629 31644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:13.619220 31644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:13.620083 31644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:13.620128 31644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:13.620170 31644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:13.620198 31644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:13.628412 31644 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:35899
I20260929 03:19:13.628477 31772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:35899 every 8 connection(s)
I20260929 03:19:13.629308 31644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:19:13.631471 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 31644
I20260929 03:19:13.631577 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance
I20260929 03:19:13.631858 14266 external_mini_cluster.cc:1428] Setting key 61b42ba52ac974e6d79fbffd2e724ad7
Sep 29 03:19:13 dist-test-slave-nbjg krb5kdc[31269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651953, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:19:13.643133 31353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:13.631141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:39633 (local address 127.13.238.190:43575)
0929 03:19:13.631307 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:13.631312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:13.632219 (+ 907us) server_negotiation.cc:407] Connection header received
0929 03:19:13.633655 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:13.633659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:13.633715 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:13.633795 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:13.635061 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.635591 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.636771 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.637082 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.637173 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:13.639668 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:13.639672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:13.639674 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:13.639701 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:13.640964 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:13.641513 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:13.641516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:13.641517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:13.641597 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:13.641919 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:13.641962 (+ 43us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:13.641963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:13.642155 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:13.642298 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:13.642823 (+ 525us) server_negotiation.cc:299] Negotiation successful
0929 03:19:13.642966 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 03:19:13.644021 31775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:13.631500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:43575 (local address 127.13.238.131:39633)
0929 03:19:13.632044 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:13.632104 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:13.633444 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:13.633996 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:13.634005 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:13.634420 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:13.634925 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:13.634940 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.635690 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.635693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:13.636632 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:13.636641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.637215 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.637217 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:13.637361 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:13.637888 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:13.637904 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:13.639474 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:13.641079 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:13.641086 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:13.641105 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:13.641397 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:13.641682 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:13.641685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:13.641688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:13.641832 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:13.642298 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:13.642305 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:13.642584 (+ 279us) client_negotiation.cc:769] Sending connection context
0929 03:19:13.642763 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 03:19:13.643122 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":112,"threads_started":1}
I20260929 03:19:13.645203 31773 heartbeater.cc:347] Connected to a master server at 127.13.238.190:43575
I20260929 03:19:13.645449 31773 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:13.645951 31773 heartbeater.cc:511] Master 127.13.238.190:43575 requested a full tablet report, sending...
I20260929 03:19:13.647071 31312 ts_manager.cc:194] Registered new tserver with Master: a09fc001b595468589711e080703abaf (127.13.238.131:35899)
I20260929 03:19:13.647627 31312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.131@KRBTEST.COM'} at 127.13.238.131:39633
I20260929 03:19:13.657572 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:19:13.666697 31353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:13.659535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41768 (local address 127.13.238.190:43575)
0929 03:19:13.659680 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:13.659706 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:13.659835 (+ 129us) server_negotiation.cc:407] Connection header received
0929 03:19:13.660003 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:13.660005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:13.660050 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:13.660128 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:13.660848 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.661366 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.662221 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.662447 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.662536 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:13.663782 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:13.663785 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:13.663787 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:13.663811 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:13.665015 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:13.665426 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:13.665429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:13.665431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:13.665509 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:13.665751 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:13.665754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:13.665756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:13.666045 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:13.666164 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:13.666429 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 03:19:13.666541 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:19:13.669569 31312 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:19:13.672739 31312 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:19:13.686581 31787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:13.681702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:35779 (local address 127.0.0.1:58596)
0929 03:19:13.681946 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:13.681958 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:13.682034 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:13.683022 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:13.683025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:13.683044 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:13.683152 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:13.683159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.684112 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.684116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:13.685267 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:13.685276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.686090 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.686093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:13.686191 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:13.686268 (+ 77us) client_negotiation.cc:769] Sending connection context
0929 03:19:13.686359 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 03:19:13.686462 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":87,"threads_started":1}
I20260929 03:19:13.686641 31789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:13.682267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58596 (local address 127.13.238.130:35779)
0929 03:19:13.682590 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:13.682603 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:13.682622 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:19:13.682686 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:13.682691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:13.682845 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:13.683033 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:13.683257 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.683995 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.685378 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.685974 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.686098 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:13.686272 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:13.686399 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 03:19:13.686499 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":107,"threads_started":1}
I20260929 03:19:13.687539 31784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:13.680971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:35899 (local address 127.0.0.1:38368)
0929 03:19:13.681524 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:13.681580 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:13.681714 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:13.682834 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:13.682840 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:13.682875 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:13.683084 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:13.683092 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.684614 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.684618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:13.686018 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:13.686028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.687138 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.687142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:13.687239 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:13.687262 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:19:13.687337 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 03:19:13.687410 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":149,"spinlock_wait_cycles":6912,"thread_start_us":100,"threads_started":1}
I20260929 03:19:13.687608 31785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:13.681156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38368 (local address 127.13.238.131:35899)
0929 03:19:13.682316 (+ 1160us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:13.682330 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:13.682354 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:19:13.682445 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:13.682451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:13.682673 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:13.682824 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:13.683212 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.684438 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.686142 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.687017 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.687140 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:13.687258 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:13.687371 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 03:19:13.687473 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1022,"thread_start_us":137,"threads_started":1}
I20260929 03:19:13.688516 31786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:13.681338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:39671 (local address 127.0.0.1:36672)
0929 03:19:13.681706 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:13.681718 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:13.681800 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:13.682836 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:13.682841 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:13.682873 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:13.683084 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:13.683092 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.684680 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.684682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:13.685974 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:13.685984 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.688145 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.688149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:13.688242 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:13.688264 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 03:19:13.688323 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 03:19:13.688388 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":95,"spinlock_wait_cycles":6400,"thread_start_us":117,"threads_started":1}
I20260929 03:19:13.689111 31788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:13.681797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36672 (local address 127.13.238.129:39671)
0929 03:19:13.682413 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:13.682428 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:13.682450 (+ 22us) server_negotiation.cc:407] Connection header received
0929 03:19:13.682539 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:13.682544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:13.682705 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:13.683229 (+ 524us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:13.683370 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.684556 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.686821 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:13.687645 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:13.687940 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:13.688071 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:13.688921 (+ 850us) server_negotiation.cc:299] Negotiation successful
0929 03:19:13.689005 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":124,"threads_started":1}
I20260929 03:19:13.690161 31573 tablet_service.cc:1467] Processing CreateTablet for tablet 9b8b20e8a37e4588a527d2d9f097f5d7 (DEFAULT_TABLE table=test-table [id=5ff321a230b247b09b6d1dcd8d0ae7b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:19:13.690613 31707 tablet_service.cc:1467] Processing CreateTablet for tablet 9b8b20e8a37e4588a527d2d9f097f5d7 (DEFAULT_TABLE table=test-table [id=5ff321a230b247b09b6d1dcd8d0ae7b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:19:13.691414 31573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b8b20e8a37e4588a527d2d9f097f5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:13.691591 31707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b8b20e8a37e4588a527d2d9f097f5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:13.691396 31439 tablet_service.cc:1467] Processing CreateTablet for tablet 9b8b20e8a37e4588a527d2d9f097f5d7 (DEFAULT_TABLE table=test-table [id=5ff321a230b247b09b6d1dcd8d0ae7b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:19:13.692302 31439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b8b20e8a37e4588a527d2d9f097f5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:13.698575 31790 tablet_bootstrap.cc:492] T 9b8b20e8a37e4588a527d2d9f097f5d7 P 63e71c018e9f4667a3fdcd5d38062989: Bootstrap starting.
I20260929 03:19:13.699641 31792 tablet_bootstrap.cc:492] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf: Bootstrap starting.
I20260929 03:19:13.700069 31791 tablet_bootstrap.cc:492] T 9b8b20e8a37e4588a527d2d9f097f5d7 P b5a278eecb104d6fa6df44f49269fe41: Bootstrap starting.
I20260929 03:19:13.701164 31790 tablet_bootstrap.cc:654] T 9b8b20e8a37e4588a527d2d9f097f5d7 P 63e71c018e9f4667a3fdcd5d38062989: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:13.701884 31790 log.cc:824] T 9b8b20e8a37e4588a527d2d9f097f5d7 P 63e71c018e9f4667a3fdcd5d38062989: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:13.702040 31791 tablet_bootstrap.cc:654] T 9b8b20e8a37e4588a527d2d9f097f5d7 P b5a278eecb104d6fa6df44f49269fe41: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:13.703284 31791 log.cc:824] T 9b8b20e8a37e4588a527d2d9f097f5d7 P b5a278eecb104d6fa6df44f49269fe41: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:13.703817 31790 tablet_bootstrap.cc:492] T 9b8b20e8a37e4588a527d2d9f097f5d7 P 63e71c018e9f4667a3fdcd5d38062989: No bootstrap required, opened a new log
I20260929 03:19:13.704041 31790 ts_tablet_manager.cc:1397] T 9b8b20e8a37e4588a527d2d9f097f5d7 P 63e71c018e9f4667a3fdcd5d38062989: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260929 03:19:13.704428 31792 tablet_bootstrap.cc:654] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:13.705415 31791 tablet_bootstrap.cc:492] T 9b8b20e8a37e4588a527d2d9f097f5d7 P b5a278eecb104d6fa6df44f49269fe41: No bootstrap required, opened a new log
I20260929 03:19:13.705598 31792 log.cc:824] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:13.705682 31791 ts_tablet_manager.cc:1397] T 9b8b20e8a37e4588a527d2d9f097f5d7 P b5a278eecb104d6fa6df44f49269fe41: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 03:19:13.707741 31790 raft_consensus.cc:348] T 9b8b20e8a37e4588a527d2d9f097f5d7 P 63e71c018e9f4667a3fdcd5d38062989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09fc001b595468589711e080703abaf" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 35899 } } peers { permanent_uuid: "b5a278eecb104d6fa6df44f49269fe41" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35779 } } peers { permanent_uuid: "63e71c018e9f4667a3fdcd5d38062989" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39671 } }
I20260929 03:19:13.707990 31790 raft_consensus.cc:374] T 9b8b20e8a37e4588a527d2d9f097f5d7 P 63e71c018e9f4667a3fdcd5d38062989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:13.708025 31790 raft_consensus.cc:729] T 9b8b20e8a37e4588a527d2d9f097f5d7 P 63e71c018e9f4667a3fdcd5d38062989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e71c018e9f4667a3fdcd5d38062989, State: Initialized, Role: FOLLOWER
I20260929 03:19:13.708449 31790 consensus_queue.cc:260] T 9b8b20e8a37e4588a527d2d9f097f5d7 P 63e71c018e9f4667a3fdcd5d38062989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09fc001b595468589711e080703abaf" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 35899 } } peers { permanent_uuid: "b5a278eecb104d6fa6df44f49269fe41" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35779 } } peers { permanent_uuid: "63e71c018e9f4667a3fdcd5d38062989" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39671 } }
I20260929 03:19:13.708732 31791 raft_consensus.cc:348] T 9b8b20e8a37e4588a527d2d9f097f5d7 P b5a278eecb104d6fa6df44f49269fe41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09fc001b595468589711e080703abaf" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 35899 } } peers { permanent_uuid: "b5a278eecb104d6fa6df44f49269fe41" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35779 } } peers { permanent_uuid: "63e71c018e9f4667a3fdcd5d38062989" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39671 } }
I20260929 03:19:13.709048 31791 raft_consensus.cc:374] T 9b8b20e8a37e4588a527d2d9f097f5d7 P b5a278eecb104d6fa6df44f49269fe41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:13.709110 31791 raft_consensus.cc:729] T 9b8b20e8a37e4588a527d2d9f097f5d7 P b5a278eecb104d6fa6df44f49269fe41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a278eecb104d6fa6df44f49269fe41, State: Initialized, Role: FOLLOWER
I20260929 03:19:13.709115 31505 heartbeater.cc:503] Master 127.13.238.190:43575 was elected leader, sending a full tablet report...
I20260929 03:19:13.709369 31790 ts_tablet_manager.cc:1428] T 9b8b20e8a37e4588a527d2d9f097f5d7 P 63e71c018e9f4667a3fdcd5d38062989: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 03:19:13.709501 31791 consensus_queue.cc:260] T 9b8b20e8a37e4588a527d2d9f097f5d7 P b5a278eecb104d6fa6df44f49269fe41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09fc001b595468589711e080703abaf" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 35899 } } peers { permanent_uuid: "b5a278eecb104d6fa6df44f49269fe41" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35779 } } peers { permanent_uuid: "63e71c018e9f4667a3fdcd5d38062989" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39671 } }
I20260929 03:19:13.710068 31639 heartbeater.cc:503] Master 127.13.238.190:43575 was elected leader, sending a full tablet report...
I20260929 03:19:13.710448 31791 ts_tablet_manager.cc:1428] T 9b8b20e8a37e4588a527d2d9f097f5d7 P b5a278eecb104d6fa6df44f49269fe41: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 03:19:13.713021 31792 tablet_bootstrap.cc:492] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf: No bootstrap required, opened a new log
I20260929 03:19:13.713244 31792 ts_tablet_manager.cc:1397] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20260929 03:19:13.716518 31792 raft_consensus.cc:348] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09fc001b595468589711e080703abaf" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 35899 } } peers { permanent_uuid: "b5a278eecb104d6fa6df44f49269fe41" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35779 } } peers { permanent_uuid: "63e71c018e9f4667a3fdcd5d38062989" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39671 } }
I20260929 03:19:13.716746 31792 raft_consensus.cc:374] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:13.716794 31792 raft_consensus.cc:729] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a09fc001b595468589711e080703abaf, State: Initialized, Role: FOLLOWER
I20260929 03:19:13.717195 31792 consensus_queue.cc:260] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09fc001b595468589711e080703abaf" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 35899 } } peers { permanent_uuid: "b5a278eecb104d6fa6df44f49269fe41" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35779 } } peers { permanent_uuid: "63e71c018e9f4667a3fdcd5d38062989" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39671 } }
I20260929 03:19:13.717808 31773 heartbeater.cc:503] Master 127.13.238.190:43575 was elected leader, sending a full tablet report...
I20260929 03:19:13.718027 31792 ts_tablet_manager.cc:1428] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260929 03:19:13.880275 31774 tablet.cc:2369] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:19:13.910104 31640 tablet.cc:2369] T 9b8b20e8a37e4588a527d2d9f097f5d7 P b5a278eecb104d6fa6df44f49269fe41: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:19:13.932497 31506 tablet.cc:2369] T 9b8b20e8a37e4588a527d2d9f097f5d7 P 63e71c018e9f4667a3fdcd5d38062989: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:19:14.054234 31798 raft_consensus.cc:482] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:19:14.054461 31798 raft_consensus.cc:504] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09fc001b595468589711e080703abaf" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 35899 } } peers { permanent_uuid: "b5a278eecb104d6fa6df44f49269fe41" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35779 } } peers { permanent_uuid: "63e71c018e9f4667a3fdcd5d38062989" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39671 } }
I20260929 03:19:14.055694 31798 leader_election.cc:288] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a278eecb104d6fa6df44f49269fe41 (127.13.238.130:35779), 63e71c018e9f4667a3fdcd5d38062989 (127.13.238.129:39671)
I20260929 03:19:14.059798 31789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:14.056103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:56601 (local address 127.13.238.130:35779)
0929 03:19:14.056236 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:14.056240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:14.056631 (+ 391us) server_negotiation.cc:407] Connection header received
0929 03:19:14.056730 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:14.056732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:14.056773 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:14.056858 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:14.057081 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:14.057818 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:14.058891 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:14.059273 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:14.059365 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:14.059416 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:14.059601 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 03:19:14.059674 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 03:19:14.059808 31799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:14.056051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:35779 (local address 127.13.238.131:56601)
0929 03:19:14.056537 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:14.056550 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:14.056655 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:14.056875 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:14.056877 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:14.056894 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:14.056987 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:14.056992 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:14.057916 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:14.057919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:14.058757 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:14.058767 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:14.059380 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:14.059384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:14.059475 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:14.059501 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 03:19:14.059578 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 03:19:14.059643 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":139,"spinlock_wait_cycles":2688,"thread_start_us":125,"threads_started":1}
I20260929 03:19:14.059900 31788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:14.056137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:40811 (local address 127.13.238.129:39671)
0929 03:19:14.056264 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:14.056269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:14.056287 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:19:14.056407 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:14.056411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:14.056467 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:14.056536 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:14.056857 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:14.057634 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:14.058908 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:14.059390 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:14.059498 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:14.059543 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:14.059718 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 03:19:14.059792 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:19:14.059943 31775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:14.056035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:39671 (local address 127.13.238.131:40811)
0929 03:19:14.056169 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:14.056185 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:14.056293 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:14.056574 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:14.056577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:14.056597 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:14.056740 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:14.056746 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:14.057814 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:14.057817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:14.058742 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:14.058753 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:14.059507 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:14.059510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:14.059600 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:14.059622 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 03:19:14.059692 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 03:19:14.059758 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"spinlock_wait_cycles":6528}
I20260929 03:19:14.060530 31593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8b20e8a37e4588a527d2d9f097f5d7" candidate_uuid: "a09fc001b595468589711e080703abaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a278eecb104d6fa6df44f49269fe41" is_pre_election: true
I20260929 03:19:14.060541 31459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8b20e8a37e4588a527d2d9f097f5d7" candidate_uuid: "a09fc001b595468589711e080703abaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63e71c018e9f4667a3fdcd5d38062989" is_pre_election: true
I20260929 03:19:14.060798 31593 raft_consensus.cc:2454] T 9b8b20e8a37e4588a527d2d9f097f5d7 P b5a278eecb104d6fa6df44f49269fe41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a09fc001b595468589711e080703abaf in term 0.
I20260929 03:19:14.060804 31459 raft_consensus.cc:2454] T 9b8b20e8a37e4588a527d2d9f097f5d7 P 63e71c018e9f4667a3fdcd5d38062989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a09fc001b595468589711e080703abaf in term 0.
I20260929 03:19:14.061223 31661 leader_election.cc:302] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a09fc001b595468589711e080703abaf, b5a278eecb104d6fa6df44f49269fe41; no voters:
I20260929 03:19:14.061473 31798 raft_consensus.cc:2781] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:19:14.061549 31798 raft_consensus.cc:482] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:19:14.061595 31798 raft_consensus.cc:3037] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:14.062716 31798 raft_consensus.cc:504] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09fc001b595468589711e080703abaf" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 35899 } } peers { permanent_uuid: "b5a278eecb104d6fa6df44f49269fe41" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35779 } } peers { permanent_uuid: "63e71c018e9f4667a3fdcd5d38062989" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39671 } }
I20260929 03:19:14.063045 31798 leader_election.cc:288] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf [CANDIDATE]: Term 1 election: Requested vote from peers b5a278eecb104d6fa6df44f49269fe41 (127.13.238.130:35779), 63e71c018e9f4667a3fdcd5d38062989 (127.13.238.129:39671)
I20260929 03:19:14.063367 31593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8b20e8a37e4588a527d2d9f097f5d7" candidate_uuid: "a09fc001b595468589711e080703abaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a278eecb104d6fa6df44f49269fe41"
I20260929 03:19:14.063380 31459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8b20e8a37e4588a527d2d9f097f5d7" candidate_uuid: "a09fc001b595468589711e080703abaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63e71c018e9f4667a3fdcd5d38062989"
I20260929 03:19:14.063472 31593 raft_consensus.cc:3037] T 9b8b20e8a37e4588a527d2d9f097f5d7 P b5a278eecb104d6fa6df44f49269fe41 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:14.063479 31459 raft_consensus.cc:3037] T 9b8b20e8a37e4588a527d2d9f097f5d7 P 63e71c018e9f4667a3fdcd5d38062989 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:14.064466 31593 raft_consensus.cc:2454] T 9b8b20e8a37e4588a527d2d9f097f5d7 P b5a278eecb104d6fa6df44f49269fe41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a09fc001b595468589711e080703abaf in term 1.
I20260929 03:19:14.064472 31459 raft_consensus.cc:2454] T 9b8b20e8a37e4588a527d2d9f097f5d7 P 63e71c018e9f4667a3fdcd5d38062989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a09fc001b595468589711e080703abaf in term 1.
I20260929 03:19:14.064754 31660 leader_election.cc:302] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63e71c018e9f4667a3fdcd5d38062989, a09fc001b595468589711e080703abaf; no voters:
I20260929 03:19:14.064903 31798 raft_consensus.cc:2781] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:14.065102 31798 raft_consensus.cc:686] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf [term 1 LEADER]: Becoming Leader. State: Replica: a09fc001b595468589711e080703abaf, State: Running, Role: LEADER
I20260929 03:19:14.065438 31798 consensus_queue.cc:237] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09fc001b595468589711e080703abaf" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 35899 } } peers { permanent_uuid: "b5a278eecb104d6fa6df44f49269fe41" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35779 } } peers { permanent_uuid: "63e71c018e9f4667a3fdcd5d38062989" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39671 } }
I20260929 03:19:14.068616 31314 catalog_manager.cc:5495] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf reported cstate change: term changed from 0 to 1, leader changed from <none> to a09fc001b595468589711e080703abaf (127.13.238.131). New cstate: current_term: 1 leader_uuid: "a09fc001b595468589711e080703abaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09fc001b595468589711e080703abaf" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 35899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5a278eecb104d6fa6df44f49269fe41" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63e71c018e9f4667a3fdcd5d38062989" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39671 } health_report { overall_health: UNKNOWN } } }
I20260929 03:19:14.141932 31785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:14.138240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38376 (local address 127.13.238.131:35899)
0929 03:19:14.138389 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:14.138393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:14.138409 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:19:14.138490 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:14.138492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:14.138551 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:14.138643 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:19:14.138891 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:14.139420 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:14.140421 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:14.140647 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:14.140733 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:14.141026 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:19:14.141570 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:19:14.141662 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:14.141761 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 03:19:14.141816 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:19:14.147730 31593 raft_consensus.cc:1261] T 9b8b20e8a37e4588a527d2d9f097f5d7 P b5a278eecb104d6fa6df44f49269fe41 [term 1 FOLLOWER]: Refusing update from remote peer a09fc001b595468589711e080703abaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:19:14.147730 31459 raft_consensus.cc:1261] T 9b8b20e8a37e4588a527d2d9f097f5d7 P 63e71c018e9f4667a3fdcd5d38062989 [term 1 FOLLOWER]: Refusing update from remote peer a09fc001b595468589711e080703abaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:19:14.148438 31800 consensus_queue.cc:1035] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "63e71c018e9f4667a3fdcd5d38062989" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 39671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:19:14.148614 31798 consensus_queue.cc:1035] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a278eecb104d6fa6df44f49269fe41" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 35779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:19:14.155903 31805 mvcc.cc:205] Tried to move back new op lower bound from 7334510404181966848 to 7334510403854688256. Current Snapshot: MvccSnapshot[applied={T|T < 7334510404181966848}]
I20260929 03:19:14.166144 31312 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:19:14.166399 31312 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:19:14.169150 31312 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 717e4cc940b54761995f76f0e55fe6cc: Sending DeleteTablet for 3 replicas of tablet 9b8b20e8a37e4588a527d2d9f097f5d7
I20260929 03:19:14.170075 31573 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b8b20e8a37e4588a527d2d9f097f5d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:19:14 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:58596
I20260929 03:19:14.170047 31439 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b8b20e8a37e4588a527d2d9f097f5d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:19:14 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:36672
I20260929 03:19:14.170145 31707 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b8b20e8a37e4588a527d2d9f097f5d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:19:14 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:38368
I20260929 03:19:14.170635 31813 tablet_replica.cc:330] T 9b8b20e8a37e4588a527d2d9f097f5d7 P b5a278eecb104d6fa6df44f49269fe41: stopping tablet replica
I20260929 03:19:14.170770 31814 tablet_replica.cc:330] T 9b8b20e8a37e4588a527d2d9f097f5d7 P 63e71c018e9f4667a3fdcd5d38062989: stopping tablet replica
I20260929 03:19:14.171132 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 31376
I20260929 03:19:14.172546 31813 raft_consensus.cc:2229] T 9b8b20e8a37e4588a527d2d9f097f5d7 P b5a278eecb104d6fa6df44f49269fe41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:19:14.173107 31813 raft_consensus.cc:2258] T 9b8b20e8a37e4588a527d2d9f097f5d7 P b5a278eecb104d6fa6df44f49269fe41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:19:14.174716 31813 ts_tablet_manager.cc:1905] T 9b8b20e8a37e4588a527d2d9f097f5d7 P b5a278eecb104d6fa6df44f49269fe41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:19:14.177299 31815 tablet_replica.cc:330] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf: stopping tablet replica
I20260929 03:19:14.177443 31813 ts_tablet_manager.cc:1918] T 9b8b20e8a37e4588a527d2d9f097f5d7 P b5a278eecb104d6fa6df44f49269fe41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:19:14.177572 31813 log.cc:1196] T 9b8b20e8a37e4588a527d2d9f097f5d7 P b5a278eecb104d6fa6df44f49269fe41: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-1/wal/wals/9b8b20e8a37e4588a527d2d9f097f5d7
I20260929 03:19:14.177838 31815 raft_consensus.cc:2229] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:19:14.177897 31813 ts_tablet_manager.cc:1939] T 9b8b20e8a37e4588a527d2d9f097f5d7 P b5a278eecb104d6fa6df44f49269fe41: Deleting consensus metadata
I20260929 03:19:14.178445 31815 raft_consensus.cc:2258] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:19:14.179286 31299 catalog_manager.cc:4841] TS b5a278eecb104d6fa6df44f49269fe41 (127.13.238.130:35779): tablet 9b8b20e8a37e4588a527d2d9f097f5d7 (table test-table [id=5ff321a230b247b09b6d1dcd8d0ae7b7]) successfully deleted
W20260929 03:19:14.179848 31298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.238.129:39671 (error 108)
I20260929 03:19:14.180029 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 31510
I20260929 03:19:14.180408 31815 ts_tablet_manager.cc:1905] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:19:14.181530 31786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:14.181178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:39671 (local address 127.0.0.1:36680)
0929 03:19:14.181368 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:14.181437 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.129:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260929 03:19:14.181715 31298 catalog_manager.cc:4584] TS 63e71c018e9f4667a3fdcd5d38062989 (127.13.238.129:39671): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9b8b20e8a37e4588a527d2d9f097f5d7: Network error: Client connection negotiation failed: client connection to 127.13.238.129:39671: connect: Connection refused (error 111)
I20260929 03:19:14.183013 31815 ts_tablet_manager.cc:1918] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:19:14.183100 31815 log.cc:1196] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/9b8b20e8a37e4588a527d2d9f097f5d7
I20260929 03:19:14.183367 31815 ts_tablet_manager.cc:1939] T 9b8b20e8a37e4588a527d2d9f097f5d7 P a09fc001b595468589711e080703abaf: Deleting consensus metadata
I20260929 03:19:14.184571 31299 catalog_manager.cc:4841] TS a09fc001b595468589711e080703abaf (127.13.238.131:35899): tablet 9b8b20e8a37e4588a527d2d9f097f5d7 (table test-table [id=5ff321a230b247b09b6d1dcd8d0ae7b7]) successfully deleted
I20260929 03:19:14.187202 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 31644
I20260929 03:19:14.192206 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 31282
2026-09-29T03:19:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2598 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T03:19:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:14Z Disabled control of system clock
I20260929 03:19:14.220325 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:42395
--webserver_interface=127.13.238.190
--webserver_port=0
--builtin_ntp_servers=127.13.238.148:46879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:42395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:19:14.330992 31820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:14.331269 31820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:14.331303 31820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:14.335255 31820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:14.335310 31820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:14.335325 31820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:14.335340 31820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:14.335353 31820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:14.340135 31820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:46879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:42395
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:42395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:14.341259 31820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:14.342665 31820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:14.350852 31825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:14.350980 31820 server_base.cc:1034] running on GCE node
W20260929 03:19:14.350886 31828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:14.350852 31826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:14.351686 31820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:14.352772 31820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:14.353927 31820 hybrid_clock.cc:648] HybridClock initialized: now 1790651954353909 us; error 27 us; skew 500 ppm
I20260929 03:19:14.355763 31820 webserver.cc:460] Webserver started at http://127.13.238.190:36549/ using document root <none> and password file <none>
I20260929 03:19:14.356326 31820 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:14.356390 31820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:14.356575 31820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:14.358790 31820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "41c8e0a524134c8d99803d6144b68a49"
format_stamp: "Formatted at 2026-09-29 03:19:14 on dist-test-slave-nbjg"
server_key: "ad6b1b559ab922b227a621df593b8b92"
server_key_iv: "a11730803b39ea5bb4bc8c3bacf1742f"
server_key_version: "encryptionkey@0"
I20260929 03:19:14.359340 31820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "41c8e0a524134c8d99803d6144b68a49"
format_stamp: "Formatted at 2026-09-29 03:19:14 on dist-test-slave-nbjg"
server_key: "ad6b1b559ab922b227a621df593b8b92"
server_key_iv: "a11730803b39ea5bb4bc8c3bacf1742f"
server_key_version: "encryptionkey@0"
I20260929 03:19:14.362944 31820 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:19:14.365451 31834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:14.366806 31820 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:19:14.366923 31820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "41c8e0a524134c8d99803d6144b68a49"
format_stamp: "Formatted at 2026-09-29 03:19:14 on dist-test-slave-nbjg"
server_key: "ad6b1b559ab922b227a621df593b8b92"
server_key_iv: "a11730803b39ea5bb4bc8c3bacf1742f"
server_key_version: "encryptionkey@0"
I20260929 03:19:14.367038 31820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:14.376662 31820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:14.377396 31820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:14.377565 31820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:14.384418 31820 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:42395
I20260929 03:19:14.384433 31886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:42395 every 8 connection(s)
I20260929 03:19:14.385363 31820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:14.385639 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 31820
I20260929 03:19:14.385743 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:19:14.386080 14266 external_mini_cluster.cc:1428] Setting key 8741317fb09308980d8c0bf57311a1b8
I20260929 03:19:14.388711 31887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:14.395083 31887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41c8e0a524134c8d99803d6144b68a49: Bootstrap starting.
I20260929 03:19:14.395500 31890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:14.387562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50926 (local address 127.13.238.190:42395)
0929 03:19:14.388197 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:14.388213 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:14.388250 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:19:14.388930 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:14.388951 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:14.389368 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:14.389651 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:14.389999 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:14.391014 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:14.392242 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:14.392724 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:14.392841 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:14.393690 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:14.393695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:14.393707 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:14.394094 (+ 387us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:14.394152 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:14.394171 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:14.394299 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:14.394453 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 03:19:14.394748 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":110,"threads_started":1}
I20260929 03:19:14.397337 31887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41c8e0a524134c8d99803d6144b68a49: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:14.398110 31887 log.cc:824] T 00000000000000000000000000000000 P 41c8e0a524134c8d99803d6144b68a49: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:14.400018 31887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41c8e0a524134c8d99803d6144b68a49: No bootstrap required, opened a new log
I20260929 03:19:14.403643 31887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41c8e0a524134c8d99803d6144b68a49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c8e0a524134c8d99803d6144b68a49" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42395 } }
I20260929 03:19:14.403926 31887 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41c8e0a524134c8d99803d6144b68a49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:14.403990 31887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41c8e0a524134c8d99803d6144b68a49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c8e0a524134c8d99803d6144b68a49, State: Initialized, Role: FOLLOWER
I20260929 03:19:14.404414 31887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41c8e0a524134c8d99803d6144b68a49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c8e0a524134c8d99803d6144b68a49" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42395 } }
I20260929 03:19:14.404549 31887 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41c8e0a524134c8d99803d6144b68a49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:14.404613 31887 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41c8e0a524134c8d99803d6144b68a49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:14.404711 31887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41c8e0a524134c8d99803d6144b68a49 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:14.405608 31887 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41c8e0a524134c8d99803d6144b68a49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c8e0a524134c8d99803d6144b68a49" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42395 } }
I20260929 03:19:14.405937 31887 leader_election.cc:302] T 00000000000000000000000000000000 P 41c8e0a524134c8d99803d6144b68a49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41c8e0a524134c8d99803d6144b68a49; no voters:
I20260929 03:19:14.406258 31887 leader_election.cc:288] T 00000000000000000000000000000000 P 41c8e0a524134c8d99803d6144b68a49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:19:14.406390 31892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41c8e0a524134c8d99803d6144b68a49 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:14.406602 31892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41c8e0a524134c8d99803d6144b68a49 [term 1 LEADER]: Becoming Leader. State: Replica: 41c8e0a524134c8d99803d6144b68a49, State: Running, Role: LEADER
I20260929 03:19:14.406966 31892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41c8e0a524134c8d99803d6144b68a49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c8e0a524134c8d99803d6144b68a49" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42395 } }
I20260929 03:19:14.407505 31887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41c8e0a524134c8d99803d6144b68a49 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:14.408538 31893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41c8e0a524134c8d99803d6144b68a49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41c8e0a524134c8d99803d6144b68a49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c8e0a524134c8d99803d6144b68a49" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42395 } } }
I20260929 03:19:14.408666 31893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41c8e0a524134c8d99803d6144b68a49 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:14.408893 31894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41c8e0a524134c8d99803d6144b68a49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41c8e0a524134c8d99803d6144b68a49. Latest consensus state: current_term: 1 leader_uuid: "41c8e0a524134c8d99803d6144b68a49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c8e0a524134c8d99803d6144b68a49" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42395 } } }
I20260929 03:19:14.408980 31894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41c8e0a524134c8d99803d6144b68a49 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:14.409126 31901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:14.412376 31901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:14.418499 31901 catalog_manager.cc:1347] Generated new cluster ID: 8f71c577e8f94cd6b36b04f48ec9f399
I20260929 03:19:14.418571 31901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:14.433705 31901 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:14.434798 31901 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:14.444139 31901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41c8e0a524134c8d99803d6144b68a49: Generated new TSK 0
I20260929 03:19:14.444936 31901 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:19:14.452330 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:0
--local_ip_for_outbound_sockets=127.13.238.129
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:42395
--builtin_ntp_servers=127.13.238.148:46879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:19:14.564965 31911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:14.565232 31911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:14.565270 31911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:14.569298 31911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:14.569352 31911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:14.569406 31911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:19:14.574234 31911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:46879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:42395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:14.575310 31911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:14.576701 31911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:14.586308 31917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:14.586314 31919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:14.586359 31911 server_base.cc:1034] running on GCE node
W20260929 03:19:14.586308 31916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:14.586896 31911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:14.587540 31911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:14.588702 31911 hybrid_clock.cc:648] HybridClock initialized: now 1790651954588673 us; error 44 us; skew 500 ppm
I20260929 03:19:14.590526 31911 webserver.cc:460] Webserver started at http://127.13.238.129:33279/ using document root <none> and password file <none>
I20260929 03:19:14.591094 31911 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:14.591157 31911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:14.591346 31911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:14.593469 31911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-0/data/instance:
uuid: "c457a2bf5c514dd2bc75b562af275465"
format_stamp: "Formatted at 2026-09-29 03:19:14 on dist-test-slave-nbjg"
server_key: "7aca6d1c3cba42c3efd9e91e7a8b7453"
server_key_iv: "7f1d9837350ee95c8337575580c62004"
server_key_version: "encryptionkey@0"
I20260929 03:19:14.594002 31911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance:
uuid: "c457a2bf5c514dd2bc75b562af275465"
format_stamp: "Formatted at 2026-09-29 03:19:14 on dist-test-slave-nbjg"
server_key: "7aca6d1c3cba42c3efd9e91e7a8b7453"
server_key_iv: "7f1d9837350ee95c8337575580c62004"
server_key_version: "encryptionkey@0"
I20260929 03:19:14.597743 31911 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:19:14.600263 31925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:14.601583 31911 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:19:14.601693 31911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "c457a2bf5c514dd2bc75b562af275465"
format_stamp: "Formatted at 2026-09-29 03:19:14 on dist-test-slave-nbjg"
server_key: "7aca6d1c3cba42c3efd9e91e7a8b7453"
server_key_iv: "7f1d9837350ee95c8337575580c62004"
server_key_version: "encryptionkey@0"
I20260929 03:19:14.601787 31911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:14.616788 31911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:14.617482 31911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:14.617630 31911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:14.618273 31911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:14.619304 31911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:14.619352 31911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:14.619393 31911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:14.619410 31911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:14.627817 31911 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:42753
I20260929 03:19:14.627867 32038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:42753 every 8 connection(s)
I20260929 03:19:14.628786 31911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:19:14.638485 31890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:14.630834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:35975 (local address 127.13.238.190:42395)
0929 03:19:14.630987 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:14.630992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:14.632111 (+ 1119us) server_negotiation.cc:407] Connection header received
0929 03:19:14.633537 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:14.633540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:14.633595 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:14.633674 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:14.634672 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:14.635217 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:14.636450 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:14.636682 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:14.636778 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:14.637974 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:14.637977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:14.637979 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:14.637994 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:14.638014 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:14.638022 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:14.638085 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:14.638272 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 03:19:14.638357 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:19:14.638931 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 31911
I20260929 03:19:14.639065 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance
I20260929 03:19:14.639328 14266 external_mini_cluster.cc:1428] Setting key 50e04736169068e9c5f3c33450a15e79
I20260929 03:19:14.639381 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:14.631264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42395 (local address 127.13.238.129:35975)
0929 03:19:14.631962 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:14.632004 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:14.633295 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:14.633843 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:14.633850 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:14.633945 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:14.634526 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:14.634542 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:14.635347 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:14.635350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:14.636313 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:14.636323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:14.636872 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:14.636875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:14.637036 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:14.637661 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:19:14.637663 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:14.637708 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:19:14.637781 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:14.638092 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:14.638098 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:14.638102 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:19:14.638223 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 03:19:14.638483 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":115,"threads_started":1}
I20260929 03:19:14.640800 32039 heartbeater.cc:347] Connected to a master server at 127.13.238.190:42395
I20260929 03:19:14.641036 32039 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:14.641336 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:0
--local_ip_for_outbound_sockets=127.13.238.130
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:42395
--builtin_ntp_servers=127.13.238.148:46879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:19:14.641569 32039 heartbeater.cc:511] Master 127.13.238.190:42395 requested a full tablet report, sending...
I20260929 03:19:14.643215 31850 ts_manager.cc:194] Registered new tserver with Master: c457a2bf5c514dd2bc75b562af275465 (127.13.238.129:42753)
I20260929 03:19:14.644738 31850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.238.129:35975
W20260929 03:19:14.751459 32042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:14.751727 32042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:14.751765 32042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:14.755728 32042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:14.755785 32042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:14.755837 32042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:19:14.760614 32042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:46879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:42395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:14.761718 32042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:14.763118 32042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:14.772519 32047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:14.772590 32050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:14.772567 32048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:14.772979 32042 server_base.cc:1034] running on GCE node
I20260929 03:19:14.773378 32042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:14.774044 32042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:14.775207 32042 hybrid_clock.cc:648] HybridClock initialized: now 1790651954775171 us; error 50 us; skew 500 ppm
I20260929 03:19:14.777038 32042 webserver.cc:460] Webserver started at http://127.13.238.130:36543/ using document root <none> and password file <none>
I20260929 03:19:14.777681 32042 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:14.777748 32042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:14.777952 32042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:14.780046 32042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-1/data/instance:
uuid: "64f966f3e0ae4ab2892174c160a070ee"
format_stamp: "Formatted at 2026-09-29 03:19:14 on dist-test-slave-nbjg"
server_key: "7e5fd593cce78eafda740a2ce46b18e9"
server_key_iv: "a54cbb470a95d8fe2eb276e70079e1cd"
server_key_version: "encryptionkey@0"
I20260929 03:19:14.780589 32042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance:
uuid: "64f966f3e0ae4ab2892174c160a070ee"
format_stamp: "Formatted at 2026-09-29 03:19:14 on dist-test-slave-nbjg"
server_key: "7e5fd593cce78eafda740a2ce46b18e9"
server_key_iv: "a54cbb470a95d8fe2eb276e70079e1cd"
server_key_version: "encryptionkey@0"
I20260929 03:19:14.784492 32042 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:19:14.787145 32056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:14.788374 32042 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:19:14.788501 32042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "64f966f3e0ae4ab2892174c160a070ee"
format_stamp: "Formatted at 2026-09-29 03:19:14 on dist-test-slave-nbjg"
server_key: "7e5fd593cce78eafda740a2ce46b18e9"
server_key_iv: "a54cbb470a95d8fe2eb276e70079e1cd"
server_key_version: "encryptionkey@0"
I20260929 03:19:14.788599 32042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:14.798942 32042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:14.799594 32042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:14.799757 32042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:14.800308 32042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:14.801214 32042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:14.801258 32042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:14.801302 32042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:14.801329 32042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:14.809702 32042 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:43095
I20260929 03:19:14.809741 32169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:43095 every 8 connection(s)
I20260929 03:19:14.810672 32042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:19:14.818161 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 32042
I20260929 03:19:14.818326 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance
I20260929 03:19:14.818686 14266 external_mini_cluster.cc:1428] Setting key 5475ffb9e6cda485f05e2006ce4132c3
I20260929 03:19:14.820278 31890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:14.812475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:36927 (local address 127.13.238.190:42395)
0929 03:19:14.812609 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:14.812613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:14.813725 (+ 1112us) server_negotiation.cc:407] Connection header received
0929 03:19:14.815246 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:14.815249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:14.815303 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:14.815380 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:14.816311 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:14.816837 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:14.817926 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:14.818279 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:14.818378 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:14.819804 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:14.819809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:14.819811 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:14.819824 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:14.819846 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:14.819855 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:14.819935 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:14.820075 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 03:19:14.820140 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:19:14.821240 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:0
--local_ip_for_outbound_sockets=127.13.238.131
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:42395
--builtin_ntp_servers=127.13.238.148:46879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:19:14.821466 32172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:14.812911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42395 (local address 127.13.238.130:36927)
0929 03:19:14.813578 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:14.813621 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:14.815018 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:14.815544 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:14.815552 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:14.815647 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:14.816192 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:14.816206 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:14.816953 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:14.816956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:14.817811 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:14.817820 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:14.818558 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:14.818563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:14.818773 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:14.819507 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:19:14.819508 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:14.819567 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:19:14.819645 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:14.819951 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:14.819957 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:14.819961 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:19:14.820058 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 03:19:14.820342 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":102,"threads_started":1}
I20260929 03:19:14.822592 32170 heartbeater.cc:347] Connected to a master server at 127.13.238.190:42395
I20260929 03:19:14.822829 32170 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:14.823369 32170 heartbeater.cc:511] Master 127.13.238.190:42395 requested a full tablet report, sending...
I20260929 03:19:14.824365 31850 ts_manager.cc:194] Registered new tserver with Master: 64f966f3e0ae4ab2892174c160a070ee (127.13.238.130:43095)
I20260929 03:19:14.824976 31850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.238.130:36927
W20260929 03:19:14.935001 32173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:14.935298 32173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:14.935361 32173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:14.939574 32173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:14.939644 32173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:14.939723 32173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:19:14.944801 32173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:46879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:42395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:14.946031 32173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:14.947716 32173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:14.958048 32179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:14.958222 32173 server_base.cc:1034] running on GCE node
W20260929 03:19:14.958048 32178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:14.958374 32181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:14.958772 32173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:14.959399 32173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:14.960564 32173 hybrid_clock.cc:648] HybridClock initialized: now 1790651954960541 us; error 34 us; skew 500 ppm
I20260929 03:19:14.962343 32173 webserver.cc:460] Webserver started at http://127.13.238.131:33005/ using document root <none> and password file <none>
I20260929 03:19:14.962909 32173 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:14.962970 32173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:14.963167 32173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:14.965197 32173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-2/data/instance:
uuid: "0061cb53803b4b7faf8f9d0b188adc01"
format_stamp: "Formatted at 2026-09-29 03:19:14 on dist-test-slave-nbjg"
server_key: "c0b2c99fe0c636e6a226d8d8e4c34c4a"
server_key_iv: "313ee8abaab2e9612462a4fb60599d6e"
server_key_version: "encryptionkey@0"
I20260929 03:19:14.965728 32173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance:
uuid: "0061cb53803b4b7faf8f9d0b188adc01"
format_stamp: "Formatted at 2026-09-29 03:19:14 on dist-test-slave-nbjg"
server_key: "c0b2c99fe0c636e6a226d8d8e4c34c4a"
server_key_iv: "313ee8abaab2e9612462a4fb60599d6e"
server_key_version: "encryptionkey@0"
I20260929 03:19:14.969427 32173 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 03:19:14.972015 32187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:14.973222 32173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:19:14.973327 32173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "0061cb53803b4b7faf8f9d0b188adc01"
format_stamp: "Formatted at 2026-09-29 03:19:14 on dist-test-slave-nbjg"
server_key: "c0b2c99fe0c636e6a226d8d8e4c34c4a"
server_key_iv: "313ee8abaab2e9612462a4fb60599d6e"
server_key_version: "encryptionkey@0"
I20260929 03:19:14.973450 32173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:14.994395 32173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:14.995157 32173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:14.995327 32173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:14.995935 32173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:14.996907 32173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:14.996953 32173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:14.996995 32173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:14.997023 32173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:15.005664 32173 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:44873
I20260929 03:19:15.005767 32300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:44873 every 8 connection(s)
I20260929 03:19:15.006669 32173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:19:15.007828 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 32173
I20260929 03:19:15.007922 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance
I20260929 03:19:15.008230 14266 external_mini_cluster.cc:1428] Setting key ea98e3b5caec1ccc880cf2f2cee96660
I20260929 03:19:15.016396 31890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:15.010134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:53903 (local address 127.13.238.190:42395)
0929 03:19:15.010265 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:15.010269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:15.010300 (+ 31us) server_negotiation.cc:407] Connection header received
0929 03:19:15.011723 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:15.011725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:15.011770 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:15.011831 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:15.012716 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.013250 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.014473 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.014710 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.014788 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:15.015948 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:15.015951 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:15.015953 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:15.015965 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:15.015983 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:15.015990 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:15.016045 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:15.016222 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 03:19:15.016274 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:19:15.017362 32303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:15.009109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42395 (local address 127.13.238.131:53903)
0929 03:19:15.010142 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:15.010224 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:15.011540 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:15.011992 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:15.011998 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:15.012086 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:15.012601 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:15.012614 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.013344 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.013346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:15.014359 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:15.014369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.014899 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.014903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:15.015050 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:15.015680 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:19:15.015682 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:15.015727 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:19:15.015795 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:15.016090 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:15.016098 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:15.016104 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:19:15.016222 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 03:19:15.016471 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":141,"threads_started":1}
I20260929 03:19:15.018548 32301 heartbeater.cc:347] Connected to a master server at 127.13.238.190:42395
I20260929 03:19:15.018797 32301 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:15.019332 32301 heartbeater.cc:511] Master 127.13.238.190:42395 requested a full tablet report, sending...
I20260929 03:19:15.020355 31850 ts_manager.cc:194] Registered new tserver with Master: 0061cb53803b4b7faf8f9d0b188adc01 (127.13.238.131:44873)
I20260929 03:19:15.021020 31850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.238.131:53903
I20260929 03:19:15.022456 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:19:15.027834 31890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:15.024346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50928 (local address 127.13.238.190:42395)
0929 03:19:15.024492 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:15.024496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:15.024648 (+ 152us) server_negotiation.cc:407] Connection header received
0929 03:19:15.024707 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:15.024710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:15.024750 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:15.024812 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:15.025022 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.025518 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.026373 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.026594 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.026677 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:15.027516 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:15.027519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:15.027522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:15.027533 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:15.027551 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:15.027559 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:15.027614 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:15.027674 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 03:19:15.027722 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 03:19:15.030877 31851 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:19:15.033885 31851 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:19:15.049230 32312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:15.042608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:43095 (local address 127.0.0.1:44532)
0929 03:19:15.043770 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:15.043786 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:15.043883 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:15.044365 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:15.044368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:15.044386 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:15.044472 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:15.044477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.045955 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.045959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:15.047585 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:15.047595 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.048652 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.048656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:15.048762 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:15.048808 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 03:19:15.048935 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 03:19:15.049069 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":150,"thread_start_us":94,"threads_started":1}
I20260929 03:19:15.049578 32317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:15.043807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55312 (local address 127.13.238.129:42753)
0929 03:19:15.044119 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:15.044127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:15.044436 (+ 309us) server_negotiation.cc:407] Connection header received
0929 03:19:15.044527 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:15.044533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:15.044688 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:15.044822 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:15.044987 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.045723 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.047687 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.048532 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.048658 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:15.048841 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:15.049312 (+ 471us) server_negotiation.cc:299] Negotiation successful
0929 03:19:15.049420 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":114,"threads_started":1}
I20260929 03:19:15.049829 32314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:15.042808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:44873 (local address 127.0.0.1:52588)
0929 03:19:15.043499 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:15.043549 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:15.043677 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:15.044234 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:15.044241 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:15.044266 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:15.044424 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:15.044432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.045884 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.045887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:15.048148 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:15.048157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.049266 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.049269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:15.049529 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:15.049553 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 03:19:15.049616 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 03:19:15.049687 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":55,"thread_start_us":80,"threads_started":1}
I20260929 03:19:15.049997 32316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:15.043387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:42753 (local address 127.0.0.1:55312)
0929 03:19:15.044241 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:15.044252 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:15.044341 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:15.044805 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:15.044808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:15.044825 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:15.044902 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:15.044907 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.046008 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.046012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:15.047541 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:15.047556 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.048974 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.048980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:15.049065 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:15.049089 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 03:19:15.049157 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 03:19:15.049232 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":141,"thread_start_us":99,"threads_started":1}
I20260929 03:19:15.050015 32313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:15.042744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44532 (local address 127.13.238.130:43095)
0929 03:19:15.043953 (+ 1209us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:15.043969 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:15.043989 (+ 20us) server_negotiation.cc:407] Connection header received
0929 03:19:15.044054 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:15.044058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:15.044226 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:15.044619 (+ 393us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:15.044755 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.045821 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.047724 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.048532 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.048657 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:15.048842 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:15.048970 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 03:19:15.049848 (+ 878us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1104,"thread_start_us":135,"threads_started":1}
I20260929 03:19:15.050369 32315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:15.043279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.13.238.131:44873)
0929 03:19:15.043743 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:15.043755 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:15.043780 (+ 25us) server_negotiation.cc:407] Connection header received
0929 03:19:15.043856 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:15.043861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:15.044073 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:15.044227 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:15.044540 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.045752 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.048276 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.048947 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.049050 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:15.049205 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:15.050080 (+ 875us) server_negotiation.cc:299] Negotiation successful
0929 03:19:15.050190 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":165,"threads_started":1}
I20260929 03:19:15.052724 32235 tablet_service.cc:1467] Processing CreateTablet for tablet 98dccd3aad3b42dbb562e33411f7094f (DEFAULT_TABLE table=test-table [id=62c6dcd53b7648be81cc82304b0909f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:19:15.052841 31973 tablet_service.cc:1467] Processing CreateTablet for tablet 98dccd3aad3b42dbb562e33411f7094f (DEFAULT_TABLE table=test-table [id=62c6dcd53b7648be81cc82304b0909f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:19:15.052841 32104 tablet_service.cc:1467] Processing CreateTablet for tablet 98dccd3aad3b42dbb562e33411f7094f (DEFAULT_TABLE table=test-table [id=62c6dcd53b7648be81cc82304b0909f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:19:15.053884 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98dccd3aad3b42dbb562e33411f7094f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:15.053885 31973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98dccd3aad3b42dbb562e33411f7094f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:15.053885 32104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98dccd3aad3b42dbb562e33411f7094f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:15.060431 32318 tablet_bootstrap.cc:492] T 98dccd3aad3b42dbb562e33411f7094f P 0061cb53803b4b7faf8f9d0b188adc01: Bootstrap starting.
I20260929 03:19:15.061827 32319 tablet_bootstrap.cc:492] T 98dccd3aad3b42dbb562e33411f7094f P c457a2bf5c514dd2bc75b562af275465: Bootstrap starting.
I20260929 03:19:15.061827 32320 tablet_bootstrap.cc:492] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee: Bootstrap starting.
I20260929 03:19:15.062899 32318 tablet_bootstrap.cc:654] T 98dccd3aad3b42dbb562e33411f7094f P 0061cb53803b4b7faf8f9d0b188adc01: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:15.064037 32319 tablet_bootstrap.cc:654] T 98dccd3aad3b42dbb562e33411f7094f P c457a2bf5c514dd2bc75b562af275465: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:15.064037 32320 tablet_bootstrap.cc:654] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:15.064124 32318 log.cc:824] T 98dccd3aad3b42dbb562e33411f7094f P 0061cb53803b4b7faf8f9d0b188adc01: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:15.064878 32320 log.cc:824] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:15.064879 32319 log.cc:824] T 98dccd3aad3b42dbb562e33411f7094f P c457a2bf5c514dd2bc75b562af275465: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:15.066421 32318 tablet_bootstrap.cc:492] T 98dccd3aad3b42dbb562e33411f7094f P 0061cb53803b4b7faf8f9d0b188adc01: No bootstrap required, opened a new log
I20260929 03:19:15.066633 32318 ts_tablet_manager.cc:1397] T 98dccd3aad3b42dbb562e33411f7094f P 0061cb53803b4b7faf8f9d0b188adc01: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 03:19:15.066648 32319 tablet_bootstrap.cc:492] T 98dccd3aad3b42dbb562e33411f7094f P c457a2bf5c514dd2bc75b562af275465: No bootstrap required, opened a new log
I20260929 03:19:15.066824 32319 ts_tablet_manager.cc:1397] T 98dccd3aad3b42dbb562e33411f7094f P c457a2bf5c514dd2bc75b562af275465: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 03:19:15.067359 32320 tablet_bootstrap.cc:492] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee: No bootstrap required, opened a new log
I20260929 03:19:15.067543 32320 ts_tablet_manager.cc:1397] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 03:19:15.069978 32319 raft_consensus.cc:348] T 98dccd3aad3b42dbb562e33411f7094f P c457a2bf5c514dd2bc75b562af275465 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f966f3e0ae4ab2892174c160a070ee" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43095 } } peers { permanent_uuid: "c457a2bf5c514dd2bc75b562af275465" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42753 } } peers { permanent_uuid: "0061cb53803b4b7faf8f9d0b188adc01" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 44873 } }
I20260929 03:19:15.070219 32319 raft_consensus.cc:374] T 98dccd3aad3b42dbb562e33411f7094f P c457a2bf5c514dd2bc75b562af275465 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:15.070272 32319 raft_consensus.cc:729] T 98dccd3aad3b42dbb562e33411f7094f P c457a2bf5c514dd2bc75b562af275465 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c457a2bf5c514dd2bc75b562af275465, State: Initialized, Role: FOLLOWER
I20260929 03:19:15.070744 32319 consensus_queue.cc:260] T 98dccd3aad3b42dbb562e33411f7094f P c457a2bf5c514dd2bc75b562af275465 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f966f3e0ae4ab2892174c160a070ee" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43095 } } peers { permanent_uuid: "c457a2bf5c514dd2bc75b562af275465" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42753 } } peers { permanent_uuid: "0061cb53803b4b7faf8f9d0b188adc01" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 44873 } }
I20260929 03:19:15.071079 32318 raft_consensus.cc:348] T 98dccd3aad3b42dbb562e33411f7094f P 0061cb53803b4b7faf8f9d0b188adc01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f966f3e0ae4ab2892174c160a070ee" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43095 } } peers { permanent_uuid: "c457a2bf5c514dd2bc75b562af275465" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42753 } } peers { permanent_uuid: "0061cb53803b4b7faf8f9d0b188adc01" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 44873 } }
I20260929 03:19:15.071323 32318 raft_consensus.cc:374] T 98dccd3aad3b42dbb562e33411f7094f P 0061cb53803b4b7faf8f9d0b188adc01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:15.071370 32318 raft_consensus.cc:729] T 98dccd3aad3b42dbb562e33411f7094f P 0061cb53803b4b7faf8f9d0b188adc01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0061cb53803b4b7faf8f9d0b188adc01, State: Initialized, Role: FOLLOWER
I20260929 03:19:15.071360 32320 raft_consensus.cc:348] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f966f3e0ae4ab2892174c160a070ee" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43095 } } peers { permanent_uuid: "c457a2bf5c514dd2bc75b562af275465" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42753 } } peers { permanent_uuid: "0061cb53803b4b7faf8f9d0b188adc01" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 44873 } }
I20260929 03:19:15.071660 32320 raft_consensus.cc:374] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:15.071717 32320 raft_consensus.cc:729] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f966f3e0ae4ab2892174c160a070ee, State: Initialized, Role: FOLLOWER
I20260929 03:19:15.071808 32319 ts_tablet_manager.cc:1428] T 98dccd3aad3b42dbb562e33411f7094f P c457a2bf5c514dd2bc75b562af275465: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 03:19:15.071844 32318 consensus_queue.cc:260] T 98dccd3aad3b42dbb562e33411f7094f P 0061cb53803b4b7faf8f9d0b188adc01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f966f3e0ae4ab2892174c160a070ee" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43095 } } peers { permanent_uuid: "c457a2bf5c514dd2bc75b562af275465" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42753 } } peers { permanent_uuid: "0061cb53803b4b7faf8f9d0b188adc01" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 44873 } }
I20260929 03:19:15.072109 32320 consensus_queue.cc:260] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f966f3e0ae4ab2892174c160a070ee" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43095 } } peers { permanent_uuid: "c457a2bf5c514dd2bc75b562af275465" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42753 } } peers { permanent_uuid: "0061cb53803b4b7faf8f9d0b188adc01" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 44873 } }
I20260929 03:19:15.072211 32039 heartbeater.cc:503] Master 127.13.238.190:42395 was elected leader, sending a full tablet report...
I20260929 03:19:15.072779 32170 heartbeater.cc:503] Master 127.13.238.190:42395 was elected leader, sending a full tablet report...
I20260929 03:19:15.073141 32320 ts_tablet_manager.cc:1428] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 03:19:15.073920 32318 ts_tablet_manager.cc:1428] T 98dccd3aad3b42dbb562e33411f7094f P 0061cb53803b4b7faf8f9d0b188adc01: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 03:19:15.074287 32301 heartbeater.cc:503] Master 127.13.238.190:42395 was elected leader, sending a full tablet report...
I20260929 03:19:15.093529 32325 raft_consensus.cc:482] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:19:15.093745 32325 raft_consensus.cc:504] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f966f3e0ae4ab2892174c160a070ee" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43095 } } peers { permanent_uuid: "c457a2bf5c514dd2bc75b562af275465" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42753 } } peers { permanent_uuid: "0061cb53803b4b7faf8f9d0b188adc01" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 44873 } }
I20260929 03:19:15.095095 32325 leader_election.cc:288] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c457a2bf5c514dd2bc75b562af275465 (127.13.238.129:42753), 0061cb53803b4b7faf8f9d0b188adc01 (127.13.238.131:44873)
I20260929 03:19:15.099750 32315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:15.095454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:34643 (local address 127.13.238.131:44873)
0929 03:19:15.095574 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:15.095577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:15.096067 (+ 490us) server_negotiation.cc:407] Connection header received
0929 03:19:15.096140 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:15.096143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:15.096197 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:15.096280 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:15.096560 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.097223 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.098626 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.099250 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.099351 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:15.099393 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:15.099542 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 03:19:15.099616 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:19:15.099750 32317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:15.095431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:50319 (local address 127.13.238.129:42753)
0929 03:19:15.095560 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:15.095564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:15.095579 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:19:15.095634 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:15.095637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:15.095686 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:15.095762 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:15.096174 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.096974 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.098532 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.099160 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.099289 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:15.099343 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:15.099544 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 03:19:15.099616 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 03:19:15.099769 32172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:15.095334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:42753 (local address 127.13.238.130:50319)
0929 03:19:15.095450 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:15.095465 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:15.095554 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:15.095800 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:15.095804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:15.095828 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:15.096018 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:15.096028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.097096 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.097099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:15.098370 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:15.098382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.099286 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.099288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:15.099394 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:15.099421 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 03:19:15.099500 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 03:19:15.099615 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":21}
I20260929 03:19:15.100346 32327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:15.095337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:44873 (local address 127.13.238.130:34643)
0929 03:19:15.095945 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:15.095957 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:15.096058 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:15.096326 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:15.096328 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:15.096347 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:15.096443 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:15.096457 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.097336 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.097339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:15.098493 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:15.098501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.099342 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.099346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:15.099445 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:15.099462 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 03:19:15.099522 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 03:19:15.099614 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":303,"thread_start_us":132,"threads_started":1}
I20260929 03:19:15.100561 31993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98dccd3aad3b42dbb562e33411f7094f" candidate_uuid: "64f966f3e0ae4ab2892174c160a070ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c457a2bf5c514dd2bc75b562af275465" is_pre_election: true
I20260929 03:19:15.100878 31993 raft_consensus.cc:2454] T 98dccd3aad3b42dbb562e33411f7094f P c457a2bf5c514dd2bc75b562af275465 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64f966f3e0ae4ab2892174c160a070ee in term 0.
I20260929 03:19:15.100934 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98dccd3aad3b42dbb562e33411f7094f" candidate_uuid: "64f966f3e0ae4ab2892174c160a070ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0061cb53803b4b7faf8f9d0b188adc01" is_pre_election: true
I20260929 03:19:15.101233 32255 raft_consensus.cc:2454] T 98dccd3aad3b42dbb562e33411f7094f P 0061cb53803b4b7faf8f9d0b188adc01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64f966f3e0ae4ab2892174c160a070ee in term 0.
I20260929 03:19:15.101356 32059 leader_election.cc:302] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f966f3e0ae4ab2892174c160a070ee, c457a2bf5c514dd2bc75b562af275465; no voters:
I20260929 03:19:15.101637 32325 raft_consensus.cc:2781] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:19:15.101712 32325 raft_consensus.cc:482] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:19:15.101743 32325 raft_consensus.cc:3037] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:15.102914 32325 raft_consensus.cc:504] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f966f3e0ae4ab2892174c160a070ee" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43095 } } peers { permanent_uuid: "c457a2bf5c514dd2bc75b562af275465" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42753 } } peers { permanent_uuid: "0061cb53803b4b7faf8f9d0b188adc01" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 44873 } }
I20260929 03:19:15.103384 32325 leader_election.cc:288] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee [CANDIDATE]: Term 1 election: Requested vote from peers c457a2bf5c514dd2bc75b562af275465 (127.13.238.129:42753), 0061cb53803b4b7faf8f9d0b188adc01 (127.13.238.131:44873)
I20260929 03:19:15.103725 32255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98dccd3aad3b42dbb562e33411f7094f" candidate_uuid: "64f966f3e0ae4ab2892174c160a070ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0061cb53803b4b7faf8f9d0b188adc01"
I20260929 03:19:15.103819 31993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98dccd3aad3b42dbb562e33411f7094f" candidate_uuid: "64f966f3e0ae4ab2892174c160a070ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c457a2bf5c514dd2bc75b562af275465"
I20260929 03:19:15.103852 32255 raft_consensus.cc:3037] T 98dccd3aad3b42dbb562e33411f7094f P 0061cb53803b4b7faf8f9d0b188adc01 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:15.103938 31993 raft_consensus.cc:3037] T 98dccd3aad3b42dbb562e33411f7094f P c457a2bf5c514dd2bc75b562af275465 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:15.104909 32255 raft_consensus.cc:2454] T 98dccd3aad3b42dbb562e33411f7094f P 0061cb53803b4b7faf8f9d0b188adc01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f966f3e0ae4ab2892174c160a070ee in term 1.
I20260929 03:19:15.104923 31993 raft_consensus.cc:2454] T 98dccd3aad3b42dbb562e33411f7094f P c457a2bf5c514dd2bc75b562af275465 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f966f3e0ae4ab2892174c160a070ee in term 1.
I20260929 03:19:15.105199 32057 leader_election.cc:302] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0061cb53803b4b7faf8f9d0b188adc01, 64f966f3e0ae4ab2892174c160a070ee; no voters:
I20260929 03:19:15.105428 32325 raft_consensus.cc:2781] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:15.105612 32325 raft_consensus.cc:686] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee [term 1 LEADER]: Becoming Leader. State: Replica: 64f966f3e0ae4ab2892174c160a070ee, State: Running, Role: LEADER
I20260929 03:19:15.105954 32325 consensus_queue.cc:237] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f966f3e0ae4ab2892174c160a070ee" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43095 } } peers { permanent_uuid: "c457a2bf5c514dd2bc75b562af275465" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42753 } } peers { permanent_uuid: "0061cb53803b4b7faf8f9d0b188adc01" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 44873 } }
I20260929 03:19:15.109144 31850 catalog_manager.cc:5495] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 64f966f3e0ae4ab2892174c160a070ee (127.13.238.130). New cstate: current_term: 1 leader_uuid: "64f966f3e0ae4ab2892174c160a070ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f966f3e0ae4ab2892174c160a070ee" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 43095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c457a2bf5c514dd2bc75b562af275465" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0061cb53803b4b7faf8f9d0b188adc01" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 44873 } health_report { overall_health: UNKNOWN } } }
I20260929 03:19:15.127270 32313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:15.123683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44542 (local address 127.13.238.130:43095)
0929 03:19:15.123793 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:15.123796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:15.123808 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:19:15.123928 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:15.123931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:15.123998 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:15.124095 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:19:15.124303 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.124796 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.125690 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:15.125924 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:15.126008 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:15.126385 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:19:15.126820 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:19:15.126912 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:15.127083 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 03:19:15.127135 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 03:19:15.130111 32040 tablet.cc:2369] T 98dccd3aad3b42dbb562e33411f7094f P c457a2bf5c514dd2bc75b562af275465: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:19:15.133345 31993 raft_consensus.cc:1261] T 98dccd3aad3b42dbb562e33411f7094f P c457a2bf5c514dd2bc75b562af275465 [term 1 FOLLOWER]: Refusing update from remote peer 64f966f3e0ae4ab2892174c160a070ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:19:15.133345 32255 raft_consensus.cc:1261] T 98dccd3aad3b42dbb562e33411f7094f P 0061cb53803b4b7faf8f9d0b188adc01 [term 1 FOLLOWER]: Refusing update from remote peer 64f966f3e0ae4ab2892174c160a070ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:19:15.134029 32328 consensus_queue.cc:1035] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0061cb53803b4b7faf8f9d0b188adc01" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 44873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:19:15.134243 32325 consensus_queue.cc:1035] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "c457a2bf5c514dd2bc75b562af275465" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:19:15.140925 32331 mvcc.cc:205] Tried to move back new op lower bound from 7334510408219090944 to 7334510408116703232. Current Snapshot: MvccSnapshot[applied={T|T < 7334510408219090944}]
I20260929 03:19:15.142784 32332 mvcc.cc:205] Tried to move back new op lower bound from 7334510408219090944 to 7334510408116703232. Current Snapshot: MvccSnapshot[applied={T|T < 7334510408219090944}]
I20260929 03:19:15.144315 32333 mvcc.cc:205] Tried to move back new op lower bound from 7334510408219090944 to 7334510408116703232. Current Snapshot: MvccSnapshot[applied={T|T < 7334510408219090944 or (T in {7334510408219090944})}]
I20260929 03:19:15.150456 31851 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:19:15.150602 31851 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:19:15.152936 31851 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41c8e0a524134c8d99803d6144b68a49: Sending DeleteTablet for 3 replicas of tablet 98dccd3aad3b42dbb562e33411f7094f
I20260929 03:19:15.153776 32235 tablet_service.cc:1514] Processing DeleteTablet for tablet 98dccd3aad3b42dbb562e33411f7094f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:19:15 UTC) from {username='slave'} at 127.0.0.1:52588
I20260929 03:19:15.153776 31973 tablet_service.cc:1514] Processing DeleteTablet for tablet 98dccd3aad3b42dbb562e33411f7094f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:19:15 UTC) from {username='slave'} at 127.0.0.1:55312
I20260929 03:19:15.154435 32104 tablet_service.cc:1514] Processing DeleteTablet for tablet 98dccd3aad3b42dbb562e33411f7094f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:19:15 UTC) from {username='slave'} at 127.0.0.1:44532
I20260929 03:19:15.154697 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 31911
I20260929 03:19:15.154923 32342 tablet_replica.cc:330] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee: stopping tablet replica
I20260929 03:19:15.155294 32342 raft_consensus.cc:2229] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:19:15.155788 32341 tablet_replica.cc:330] T 98dccd3aad3b42dbb562e33411f7094f P 0061cb53803b4b7faf8f9d0b188adc01: stopping tablet replica
I20260929 03:19:15.155912 32342 raft_consensus.cc:2258] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:19:15.156070 32341 raft_consensus.cc:2229] T 98dccd3aad3b42dbb562e33411f7094f P 0061cb53803b4b7faf8f9d0b188adc01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:19:15.156476 32341 raft_consensus.cc:2258] T 98dccd3aad3b42dbb562e33411f7094f P 0061cb53803b4b7faf8f9d0b188adc01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:19:15.157722 32342 ts_tablet_manager.cc:1905] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:19:15.158161 32341 ts_tablet_manager.cc:1905] T 98dccd3aad3b42dbb562e33411f7094f P 0061cb53803b4b7faf8f9d0b188adc01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:19:15.160221 32341 ts_tablet_manager.cc:1918] T 98dccd3aad3b42dbb562e33411f7094f P 0061cb53803b4b7faf8f9d0b188adc01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:19:15.160310 32341 log.cc:1196] T 98dccd3aad3b42dbb562e33411f7094f P 0061cb53803b4b7faf8f9d0b188adc01: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/98dccd3aad3b42dbb562e33411f7094f
I20260929 03:19:15.160559 32342 ts_tablet_manager.cc:1918] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:19:15.160633 32341 ts_tablet_manager.cc:1939] T 98dccd3aad3b42dbb562e33411f7094f P 0061cb53803b4b7faf8f9d0b188adc01: Deleting consensus metadata
I20260929 03:19:15.160658 32342 log.cc:1196] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790651713532531-14266-0/minicluster-data/ts-1/wal/wals/98dccd3aad3b42dbb562e33411f7094f
I20260929 03:19:15.160939 32342 ts_tablet_manager.cc:1939] T 98dccd3aad3b42dbb562e33411f7094f P 64f966f3e0ae4ab2892174c160a070ee: Deleting consensus metadata
I20260929 03:19:15.161933 31835 catalog_manager.cc:4841] TS 0061cb53803b4b7faf8f9d0b188adc01 (127.13.238.131:44873): tablet 98dccd3aad3b42dbb562e33411f7094f (table test-table [id=62c6dcd53b7648be81cc82304b0909f4]) successfully deleted
I20260929 03:19:15.162292 31837 catalog_manager.cc:4841] TS 64f966f3e0ae4ab2892174c160a070ee (127.13.238.130:43095): tablet 98dccd3aad3b42dbb562e33411f7094f (table test-table [id=62c6dcd53b7648be81cc82304b0909f4]) successfully deleted
W20260929 03:19:15.165194 31837 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.238.129:42753 (error 108)
I20260929 03:19:15.165273 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 32042
I20260929 03:19:15.166826 32316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:15.166503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:42753 (local address 127.0.0.1:55326)
0929 03:19:15.166655 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:15.166733 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.129:42753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 03:19:15.167047 31837 catalog_manager.cc:4584] TS c457a2bf5c514dd2bc75b562af275465 (127.13.238.129:42753): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 98dccd3aad3b42dbb562e33411f7094f: Network error: Client connection negotiation failed: client connection to 127.13.238.129:42753: connect: Connection refused (error 111)
I20260929 03:19:15.170921 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 32173
I20260929 03:19:15.175673 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 31820
2026-09-29T03:19:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (983 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:19:15 dist-test-slave-nbjg krb5kdc[32346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:19:15 dist-test-slave-nbjg krb5kdc[32346](info): set up 2 sockets
Sep 29 03:19:15 dist-test-slave-nbjg krb5kdc[32346](info): commencing operation
krb5kdc: starting...
W20260929 03:19:16.134146 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:19:16 dist-test-slave-nbjg krb5kdc[32346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:19:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:19:16.253177 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:38451
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:38451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:16.363417 32359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:16.363698 32359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:16.363755 32359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:16.367791 32359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:16.367861 32359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:16.367887 32359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:16.367910 32359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:16.367936 32359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:16.373512 32359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:38451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:38451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:16.374719 32359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:16.375785 32359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:16.384660 32367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:16.384912 32359 server_base.cc:1034] running on GCE node
W20260929 03:19:16.384807 32365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:16.384690 32364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:16.385628 32359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:16.386776 32359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:16.387938 32359 hybrid_clock.cc:648] HybridClock initialized: now 1790651956387918 us; error 31 us; skew 500 ppm
Sep 29 03:19:16 dist-test-slave-nbjg krb5kdc[32346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651956, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:16.390730 32359 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:19:16.391642 32359 webserver.cc:460] Webserver started at http://127.13.238.190:32939/ using document root <none> and password file <none>
I20260929 03:19:16.392170 32359 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:16.392227 32359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:16.392400 32359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:16.394521 32359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "fbf9cde9851048dcafde5dd82bcd19f5"
format_stamp: "Formatted at 2026-09-29 03:19:16 on dist-test-slave-nbjg"
server_key: "efb825489e1fb2f5908c083f38e8af0b"
server_key_iv: "f85a071c292876b2c4b54a837e0bceb6"
server_key_version: "encryptionkey@0"
I20260929 03:19:16.395064 32359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "fbf9cde9851048dcafde5dd82bcd19f5"
format_stamp: "Formatted at 2026-09-29 03:19:16 on dist-test-slave-nbjg"
server_key: "efb825489e1fb2f5908c083f38e8af0b"
server_key_iv: "f85a071c292876b2c4b54a837e0bceb6"
server_key_version: "encryptionkey@0"
I20260929 03:19:16.398806 32359 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 03:19:16.401300 32374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:16.402617 32359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:19:16.402726 32359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "fbf9cde9851048dcafde5dd82bcd19f5"
format_stamp: "Formatted at 2026-09-29 03:19:16 on dist-test-slave-nbjg"
server_key: "efb825489e1fb2f5908c083f38e8af0b"
server_key_iv: "f85a071c292876b2c4b54a837e0bceb6"
server_key_version: "encryptionkey@0"
I20260929 03:19:16.402832 32359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:16.412523 32359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:16.415580 32359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:16.415753 32359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:16.422551 32359 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:38451
I20260929 03:19:16.422600 32426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:38451 every 8 connection(s)
I20260929 03:19:16.423496 32359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:16.426304 32427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:16.429580 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 32359
I20260929 03:19:16.429687 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:19:16.429963 14266 external_mini_cluster.cc:1428] Setting key c5920f62b43598dfbaa6221512c28521
I20260929 03:19:16.432360 32427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbf9cde9851048dcafde5dd82bcd19f5: Bootstrap starting.
I20260929 03:19:16.435142 32427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbf9cde9851048dcafde5dd82bcd19f5: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:16.436029 32427 log.cc:824] T 00000000000000000000000000000000 P fbf9cde9851048dcafde5dd82bcd19f5: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:16.438182 32427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbf9cde9851048dcafde5dd82bcd19f5: No bootstrap required, opened a new log
Sep 29 03:19:16 dist-test-slave-nbjg krb5kdc[32346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:19:16.441860 32427 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbf9cde9851048dcafde5dd82bcd19f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf9cde9851048dcafde5dd82bcd19f5" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 38451 } }
I20260929 03:19:16.442185 32427 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbf9cde9851048dcafde5dd82bcd19f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:16.442273 32427 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbf9cde9851048dcafde5dd82bcd19f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbf9cde9851048dcafde5dd82bcd19f5, State: Initialized, Role: FOLLOWER
I20260929 03:19:16.442823 32427 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbf9cde9851048dcafde5dd82bcd19f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf9cde9851048dcafde5dd82bcd19f5" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 38451 } }
I20260929 03:19:16.442965 32427 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbf9cde9851048dcafde5dd82bcd19f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:16.443030 32427 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbf9cde9851048dcafde5dd82bcd19f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:16.443137 32427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbf9cde9851048dcafde5dd82bcd19f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:16.444394 32427 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbf9cde9851048dcafde5dd82bcd19f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf9cde9851048dcafde5dd82bcd19f5" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 38451 } }
I20260929 03:19:16.444793 32427 leader_election.cc:302] T 00000000000000000000000000000000 P fbf9cde9851048dcafde5dd82bcd19f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbf9cde9851048dcafde5dd82bcd19f5; no voters:
I20260929 03:19:16.445174 32427 leader_election.cc:288] T 00000000000000000000000000000000 P fbf9cde9851048dcafde5dd82bcd19f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:19:16.445295 32432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbf9cde9851048dcafde5dd82bcd19f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:16.445518 32432 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbf9cde9851048dcafde5dd82bcd19f5 [term 1 LEADER]: Becoming Leader. State: Replica: fbf9cde9851048dcafde5dd82bcd19f5, State: Running, Role: LEADER
I20260929 03:19:16.445904 32432 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbf9cde9851048dcafde5dd82bcd19f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf9cde9851048dcafde5dd82bcd19f5" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 38451 } }
I20260929 03:19:16.446050 32430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:16.431469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51198 (local address 127.13.238.190:38451)
0929 03:19:16.431912 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:16.431932 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:16.431988 (+ 56us) server_negotiation.cc:407] Connection header received
0929 03:19:16.432792 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:16.432818 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:16.433251 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:16.433634 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:16.434362 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:16.435824 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:16.436740 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:16.437412 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:16.437569 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:16.440029 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:16.440036 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:16.440054 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:16.440088 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:16.442611 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:16.443151 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:16.443158 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:16.443166 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:16.443316 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:16.443575 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:16.443579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:16.443580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:16.444156 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:16.444364 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:16.444590 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 03:19:16.444914 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":126,"threads_started":1}
I20260929 03:19:16.446596 32427 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbf9cde9851048dcafde5dd82bcd19f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:16.447310 32433 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbf9cde9851048dcafde5dd82bcd19f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbf9cde9851048dcafde5dd82bcd19f5. Latest consensus state: current_term: 1 leader_uuid: "fbf9cde9851048dcafde5dd82bcd19f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf9cde9851048dcafde5dd82bcd19f5" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 38451 } } }
I20260929 03:19:16.447434 32433 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbf9cde9851048dcafde5dd82bcd19f5 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:16.448179 32434 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbf9cde9851048dcafde5dd82bcd19f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbf9cde9851048dcafde5dd82bcd19f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf9cde9851048dcafde5dd82bcd19f5" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 38451 } } }
I20260929 03:19:16.448333 32434 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbf9cde9851048dcafde5dd82bcd19f5 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:16.448653 32441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:16.451968 32441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:16.458598 32441 catalog_manager.cc:1347] Generated new cluster ID: d9689fa21ea04d6d8b77f0bbf96bd68a
I20260929 03:19:16.458702 32441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:16.466601 32441 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:16.467538 32441 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:16.472911 32441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbf9cde9851048dcafde5dd82bcd19f5: Generated new TSK 0
I20260929 03:19:16.473619 32441 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
I20260929 03:19:16.508951 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:0
--local_ip_for_outbound_sockets=127.13.238.129
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:38451
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:16.650238 32453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:16.650504 32453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:16.650548 32453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:16.654600 32453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:16.654663 32453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:16.654721 32453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:19:16.660141 32453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:38451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:16.661207 32453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:16.662273 32453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:16.672037 32461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:16.672042 32459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:16.672071 32458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:16.672223 32453 server_base.cc:1034] running on GCE node
I20260929 03:19:16.672816 32453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:16.673470 32453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:16.674652 32453 hybrid_clock.cc:648] HybridClock initialized: now 1790651956674626 us; error 34 us; skew 500 ppm
Sep 29 03:19:16 dist-test-slave-nbjg krb5kdc[32346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651956, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:16.681059 32453 init.cc:377] Logged in from keytab as kudu/127.13.238.129@KRBTEST.COM (short username kudu)
I20260929 03:19:16.682068 32453 webserver.cc:460] Webserver started at http://127.13.238.129:35149/ using document root <none> and password file <none>
I20260929 03:19:16.682678 32453 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:16.682749 32453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:16.682945 32453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:16.685030 32453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-0/data/instance:
uuid: "ffb3517c69464fbca1d0740b5da4b2e8"
format_stamp: "Formatted at 2026-09-29 03:19:16 on dist-test-slave-nbjg"
server_key: "d0d9d987620995cda83c74edad0209fe"
server_key_iv: "b7f274c1302ad2cd1d6c060473e61843"
server_key_version: "encryptionkey@0"
I20260929 03:19:16.685595 32453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance:
uuid: "ffb3517c69464fbca1d0740b5da4b2e8"
format_stamp: "Formatted at 2026-09-29 03:19:16 on dist-test-slave-nbjg"
server_key: "d0d9d987620995cda83c74edad0209fe"
server_key_iv: "b7f274c1302ad2cd1d6c060473e61843"
server_key_version: "encryptionkey@0"
I20260929 03:19:16.689405 32453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:19:16.692024 32468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:16.693235 32453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 03:19:16.693347 32453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "ffb3517c69464fbca1d0740b5da4b2e8"
format_stamp: "Formatted at 2026-09-29 03:19:16 on dist-test-slave-nbjg"
server_key: "d0d9d987620995cda83c74edad0209fe"
server_key_iv: "b7f274c1302ad2cd1d6c060473e61843"
server_key_version: "encryptionkey@0"
I20260929 03:19:16.693447 32453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:16.711640 32453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:16.714874 32453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:16.715065 32453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:16.715674 32453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:16.716519 32453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:16.716565 32453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:16.716607 32453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:16.716635 32453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:16.725486 32453 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:40779
I20260929 03:19:16.725528 32581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:40779 every 8 connection(s)
I20260929 03:19:16.726475 32453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
Sep 29 03:19:16 dist-test-slave-nbjg krb5kdc[32346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651956, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:19:16.736624 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 32453
I20260929 03:19:16.736758 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance
I20260929 03:19:16.737087 14266 external_mini_cluster.cc:1428] Setting key faf3f3ad4823bfe782165ec7872823d4
I20260929 03:19:16.739758 32430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:16.728363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:40963 (local address 127.13.238.190:38451)
0929 03:19:16.728558 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:16.728563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:16.729428 (+ 865us) server_negotiation.cc:407] Connection header received
0929 03:19:16.730806 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:16.730810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:16.730860 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:16.730925 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:16.732065 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:16.732596 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:16.733673 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:16.733889 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:16.733957 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:16.736309 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:16.736312 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:16.736314 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:16.736336 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:16.737746 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:16.738237 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:16.738241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:16.738242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:16.738318 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:16.738708 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:16.738710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:16.738712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:16.738884 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:16.738954 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:16.739541 (+ 587us) server_negotiation.cc:299] Negotiation successful
0929 03:19:16.739639 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 03:19:16.740810 32584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:16.728802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:38451 (local address 127.13.238.129:40963)
0929 03:19:16.729269 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:16.729316 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:16.730582 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:16.731073 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:16.731080 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:16.731436 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:16.731934 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:16.731948 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:16.732686 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:16.732688 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:16.733584 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:16.733593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:16.733999 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:16.734001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:16.734137 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:16.734732 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:16.734748 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:16.736164 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:16.737849 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:16.737855 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:16.737871 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:16.738124 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:16.738414 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:16.738417 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:16.738419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:16.738606 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:16.738979 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:16.738986 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:16.739310 (+ 324us) client_negotiation.cc:769] Sending connection context
0929 03:19:16.739519 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 03:19:16.739853 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":110,"threads_started":1}
I20260929 03:19:16.742064 32582 heartbeater.cc:347] Connected to a master server at 127.13.238.190:38451
I20260929 03:19:16.742349 32582 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:16.742934 32582 heartbeater.cc:511] Master 127.13.238.190:38451 requested a full tablet report, sending...
I20260929 03:19:16.744570 32391 ts_manager.cc:194] Registered new tserver with Master: ffb3517c69464fbca1d0740b5da4b2e8 (127.13.238.129:40779)
I20260929 03:19:16.745708 32391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.129@KRBTEST.COM'} at 127.13.238.129:40963
WARNING: no policy specified for kudu/127.13.238.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
I20260929 03:19:16.765103 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:0
--local_ip_for_outbound_sockets=127.13.238.130
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:38451
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:16.874991 32587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:16.875258 32587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:16.875319 32587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:16.879446 32587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:16.879513 32587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:16.879595 32587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:19:16.885145 32587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:38451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:16.886296 32587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:16.887332 32587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:16.897035 32595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:16.897085 32593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:16.897102 32592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:16.897310 32587 server_base.cc:1034] running on GCE node
I20260929 03:19:16.897773 32587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:16.898475 32587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:16.899649 32587 hybrid_clock.cc:648] HybridClock initialized: now 1790651956899623 us; error 37 us; skew 500 ppm
Sep 29 03:19:16 dist-test-slave-nbjg krb5kdc[32346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651956, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:16.902582 32587 init.cc:377] Logged in from keytab as kudu/127.13.238.130@KRBTEST.COM (short username kudu)
I20260929 03:19:16.903486 32587 webserver.cc:460] Webserver started at http://127.13.238.130:37751/ using document root <none> and password file <none>
I20260929 03:19:16.904047 32587 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:16.904104 32587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:16.904304 32587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:16.906375 32587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-1/data/instance:
uuid: "5dbd39b8dd4d4d9a8e5e00bd10f6bd0b"
format_stamp: "Formatted at 2026-09-29 03:19:16 on dist-test-slave-nbjg"
server_key: "ff02be975e9d6735a8e30ad9b15b5bc0"
server_key_iv: "a07bfc44b9456bc2c6b1dca627a842c5"
server_key_version: "encryptionkey@0"
I20260929 03:19:16.906967 32587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance:
uuid: "5dbd39b8dd4d4d9a8e5e00bd10f6bd0b"
format_stamp: "Formatted at 2026-09-29 03:19:16 on dist-test-slave-nbjg"
server_key: "ff02be975e9d6735a8e30ad9b15b5bc0"
server_key_iv: "a07bfc44b9456bc2c6b1dca627a842c5"
server_key_version: "encryptionkey@0"
I20260929 03:19:16.910907 32587 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 03:19:16.913511 32602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:16.914906 32587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:19:16.915020 32587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "5dbd39b8dd4d4d9a8e5e00bd10f6bd0b"
format_stamp: "Formatted at 2026-09-29 03:19:16 on dist-test-slave-nbjg"
server_key: "ff02be975e9d6735a8e30ad9b15b5bc0"
server_key_iv: "a07bfc44b9456bc2c6b1dca627a842c5"
server_key_version: "encryptionkey@0"
I20260929 03:19:16.915125 32587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:16.941249 32587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:16.944540 32587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:16.944720 32587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:16.945478 32587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:16.946388 32587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:16.946434 32587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:16.946475 32587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:16.946494 32587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:16.955152 32587 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:33931
I20260929 03:19:16.955180 32715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:33931 every 8 connection(s)
I20260929 03:19:16.956030 32587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:19:16.961484 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 32587
I20260929 03:19:16.961622 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance
I20260929 03:19:16.961916 14266 external_mini_cluster.cc:1428] Setting key d52894bd74b74d1f82c920f39b7171ea
Sep 29 03:19:16 dist-test-slave-nbjg krb5kdc[32346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651956, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:19:16.969599 32430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:16.957856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:60865 (local address 127.13.238.190:38451)
0929 03:19:16.958088 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:16.958093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:16.958911 (+ 818us) server_negotiation.cc:407] Connection header received
0929 03:19:16.960199 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:16.960201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:16.960250 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:16.960322 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:16.961475 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:16.962295 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:16.963590 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:16.963906 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:16.964009 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:16.966311 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:16.966314 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:16.966316 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:16.966340 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:16.967695 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:16.968170 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:16.968174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:16.968176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:16.968252 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:16.968562 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:16.968566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:16.968568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:16.968744 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:16.968827 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:16.969310 (+ 483us) server_negotiation.cc:299] Negotiation successful
0929 03:19:16.969434 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
WARNING: no policy specified for kudu/127.13.238.131@KRBTEST.COM; defaulting to no policy
I20260929 03:19:16.970460 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:16.958318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:38451 (local address 127.13.238.130:60865)
0929 03:19:16.958761 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:16.958804 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:16.959998 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:16.960475 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:16.960482 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:16.960842 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:16.961345 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:16.961358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:16.962423 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:16.962427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:16.963468 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:16.963478 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:16.964056 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:16.964059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:16.964199 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:16.964727 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:16.964744 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:16.966144 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:16.967801 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:16.967807 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:16.967824 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:16.968081 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:16.968330 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:16.968333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:16.968335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:16.968472 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:16.968840 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:16.968847 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:16.969088 (+ 241us) client_negotiation.cc:769] Sending connection context
0929 03:19:16.969252 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 03:19:16.969583 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":102,"threads_started":1}
I20260929 03:19:16.971581 32716 heartbeater.cc:347] Connected to a master server at 127.13.238.190:38451
I20260929 03:19:16.971828 32716 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:16.972407 32716 heartbeater.cc:511] Master 127.13.238.190:38451 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.131@KRBTEST.COM" created.
I20260929 03:19:16.973596 32391 ts_manager.cc:194] Registered new tserver with Master: 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b (127.13.238.130:33931)
I20260929 03:19:16.974160 32391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.130@KRBTEST.COM'} at 127.13.238.130:60865
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
I20260929 03:19:16.989840 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:0
--local_ip_for_outbound_sockets=127.13.238.131
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:38451
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:17.101934 32721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:17.102254 32721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:17.102315 32721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:17.106477 32721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:17.106539 32721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:17.106606 32721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:19:17.112123 32721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:38451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:17.113322 32721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:17.114455 32721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:17.124279 32727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:17.124279 32726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:17.124418 32721 server_base.cc:1034] running on GCE node
W20260929 03:19:17.124279 32729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:17.125049 32721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:17.125679 32721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:17.126855 32721 hybrid_clock.cc:648] HybridClock initialized: now 1790651957126827 us; error 36 us; skew 500 ppm
Sep 29 03:19:17 dist-test-slave-nbjg krb5kdc[32346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651957, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:17.129583 32721 init.cc:377] Logged in from keytab as kudu/127.13.238.131@KRBTEST.COM (short username kudu)
I20260929 03:19:17.130471 32721 webserver.cc:460] Webserver started at http://127.13.238.131:33389/ using document root <none> and password file <none>
I20260929 03:19:17.131014 32721 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:17.131079 32721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:17.131266 32721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:17.133289 32721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-2/data/instance:
uuid: "aab1ea6e786b4d2eb9dfb3ed31c0af06"
format_stamp: "Formatted at 2026-09-29 03:19:17 on dist-test-slave-nbjg"
server_key: "ca949a97d8ad977038c1019301ce9637"
server_key_iv: "9214c43c50513e032ef9a50d6fef00d0"
server_key_version: "encryptionkey@0"
I20260929 03:19:17.133824 32721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance:
uuid: "aab1ea6e786b4d2eb9dfb3ed31c0af06"
format_stamp: "Formatted at 2026-09-29 03:19:17 on dist-test-slave-nbjg"
server_key: "ca949a97d8ad977038c1019301ce9637"
server_key_iv: "9214c43c50513e032ef9a50d6fef00d0"
server_key_version: "encryptionkey@0"
I20260929 03:19:17.137661 32721 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:19:17.140206 32736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:17.141561 32721 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:19:17.141685 32721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "aab1ea6e786b4d2eb9dfb3ed31c0af06"
format_stamp: "Formatted at 2026-09-29 03:19:17 on dist-test-slave-nbjg"
server_key: "ca949a97d8ad977038c1019301ce9637"
server_key_iv: "9214c43c50513e032ef9a50d6fef00d0"
server_key_version: "encryptionkey@0"
I20260929 03:19:17.141795 32721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:17.161880 32721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:17.165076 32721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:17.165262 32721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:17.165879 32721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:17.166739 32721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:17.166785 32721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:17.166839 32721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:17.166865 32721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:17.175212 32721 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:43959
I20260929 03:19:17.175242 381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:43959 every 8 connection(s)
I20260929 03:19:17.176294 32721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:19:17.177665 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 32721
I20260929 03:19:17.177803 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance
I20260929 03:19:17.178135 14266 external_mini_cluster.cc:1428] Setting key e0beb0bdf287bd5a12eb2bb92be4bc1d
Sep 29 03:19:17 dist-test-slave-nbjg krb5kdc[32346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651957, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:19:17.189955 32430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:17.178426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:36849 (local address 127.13.238.190:38451)
0929 03:19:17.178597 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:17.178601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:17.179637 (+ 1036us) server_negotiation.cc:407] Connection header received
0929 03:19:17.181126 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:17.181128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:17.181175 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:17.181237 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:17.182440 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.182977 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.184070 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.184280 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.184358 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:17.186806 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:17.186809 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:17.186811 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:17.186836 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:17.188024 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:17.188491 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:17.188494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:17.188496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:17.188576 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:17.188893 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:17.188895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:17.188896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:17.189072 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:17.189155 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:17.189668 (+ 513us) server_negotiation.cc:299] Negotiation successful
0929 03:19:17.189792 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 03:19:17.190809 384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:17.178940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:38451 (local address 127.13.238.131:36849)
0929 03:19:17.179461 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:17.179530 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:17.180925 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:17.181375 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:17.181382 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:17.181736 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:17.182310 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:17.182326 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.183073 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.183076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:17.183965 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:17.183974 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.184446 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.184449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:17.184599 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:17.185140 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:17.185163 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:17.186651 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:17.188131 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:17.188137 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:17.188152 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:17.188391 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:17.188655 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:17.188659 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:17.188661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:17.188799 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:17.189165 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:17.189171 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:17.189416 (+ 245us) client_negotiation.cc:769] Sending connection context
0929 03:19:17.189571 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 03:19:17.189892 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":117,"threads_started":1}
I20260929 03:19:17.191980 382 heartbeater.cc:347] Connected to a master server at 127.13.238.190:38451
I20260929 03:19:17.192216 382 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:17.192684 382 heartbeater.cc:511] Master 127.13.238.190:38451 requested a full tablet report, sending...
I20260929 03:19:17.193678 32391 ts_manager.cc:194] Registered new tserver with Master: aab1ea6e786b4d2eb9dfb3ed31c0af06 (127.13.238.131:43959)
I20260929 03:19:17.194264 32391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.131@KRBTEST.COM'} at 127.13.238.131:36849
I20260929 03:19:17.204291 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:19:17.214226 32430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:17.206634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51200 (local address 127.13.238.190:38451)
0929 03:19:17.206800 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:17.206804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:17.206968 (+ 164us) server_negotiation.cc:407] Connection header received
0929 03:19:17.207134 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:17.207137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:17.207208 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:17.207278 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:17.208204 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.208722 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.209531 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.209739 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.209804 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:17.211378 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:17.211381 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:17.211383 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:17.211405 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:17.212503 (+ 1098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:17.212968 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:17.212970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:17.212972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:17.213048 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:17.213358 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:17.213360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:17.213362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:17.213555 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:17.213662 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:17.213907 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 03:19:17.213989 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:19:17.217314 32391 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:19:17.220479 32391 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:19:17.235741 394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:17.229568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51150 (local address 127.13.238.129:40779)
0929 03:19:17.230578 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:17.230594 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:17.230619 (+ 25us) server_negotiation.cc:407] Connection header received
0929 03:19:17.230700 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:17.230705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:17.230930 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:17.231088 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:17.231394 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.232610 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.234226 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.235071 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.235197 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:17.235333 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:17.235525 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 03:19:17.235609 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":890,"thread_start_us":67,"threads_started":1}
I20260929 03:19:17.235759 393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:17.229393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:40779 (local address 127.0.0.1:51150)
0929 03:19:17.229749 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:17.229805 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:17.229927 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:17.231101 (+ 1174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:17.231108 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:17.231129 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:17.231281 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:17.231289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.232742 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.232746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:17.234097 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:17.234108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.235192 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.235195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:17.235315 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:17.235368 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 03:19:17.235480 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 03:19:17.235607 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":88,"thread_start_us":75,"threads_started":1}
I20260929 03:19:17.236145 396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:17.230119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:43959 (local address 127.0.0.1:39920)
0929 03:19:17.230999 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:17.231011 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:17.231103 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:17.231507 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:17.231510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:17.231528 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:17.231611 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:17.231617 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.232882 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.232885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:17.234777 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:17.234787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.235781 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.235785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:17.235872 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:17.235896 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 03:19:17.235957 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 03:19:17.236023 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":131,"thread_start_us":117,"threads_started":1}
I20260929 03:19:17.236423 398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:17.230587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39920 (local address 127.13.238.131:43959)
0929 03:19:17.230966 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:17.230978 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:17.231086 (+ 108us) server_negotiation.cc:407] Connection header received
0929 03:19:17.231210 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:17.231215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:17.231367 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:17.231537 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:17.231706 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.232759 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.234901 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.235662 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.235777 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:17.236079 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:17.236131 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 03:19:17.236289 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":164,"threads_started":1}
I20260929 03:19:17.237514 395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:17.229643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:33931 (local address 127.0.0.1:36394)
0929 03:19:17.230244 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:17.230256 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:17.230337 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:17.231763 (+ 1426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:17.231765 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:17.231781 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:17.231869 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:17.231875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.232819 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.232823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:17.234055 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:17.234064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.236385 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.236389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:17.236479 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:17.237228 (+ 749us) client_negotiation.cc:769] Sending connection context
0929 03:19:17.237309 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 03:19:17.237391 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":68,"thread_start_us":122,"threads_started":1}
I20260929 03:19:17.237591 397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:17.230233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36394 (local address 127.13.238.130:33931)
0929 03:19:17.231366 (+ 1133us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:17.231379 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:17.231400 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:19:17.231467 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:17.231473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:17.231654 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:17.231798 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:17.231943 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.232685 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.234167 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.235071 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.235206 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:17.235333 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:17.237358 (+ 2025us) server_negotiation.cc:299] Negotiation successful
0929 03:19:17.237459 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1015,"thread_start_us":114,"threads_started":1}
I20260929 03:19:17.239037 32516 tablet_service.cc:1467] Processing CreateTablet for tablet ea6a1603bb2946a7a80df3f0887b22a2 (DEFAULT_TABLE table=test-table [id=70cae6c7611342e69a8ed7529278978d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:19:17.239182 316 tablet_service.cc:1467] Processing CreateTablet for tablet ea6a1603bb2946a7a80df3f0887b22a2 (DEFAULT_TABLE table=test-table [id=70cae6c7611342e69a8ed7529278978d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:19:17.239980 32516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea6a1603bb2946a7a80df3f0887b22a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:17.240188 316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea6a1603bb2946a7a80df3f0887b22a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:17.240118 32650 tablet_service.cc:1467] Processing CreateTablet for tablet ea6a1603bb2946a7a80df3f0887b22a2 (DEFAULT_TABLE table=test-table [id=70cae6c7611342e69a8ed7529278978d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:19:17.241050 32650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea6a1603bb2946a7a80df3f0887b22a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:17.246234 399 tablet_bootstrap.cc:492] T ea6a1603bb2946a7a80df3f0887b22a2 P ffb3517c69464fbca1d0740b5da4b2e8: Bootstrap starting.
I20260929 03:19:17.248142 400 tablet_bootstrap.cc:492] T ea6a1603bb2946a7a80df3f0887b22a2 P aab1ea6e786b4d2eb9dfb3ed31c0af06: Bootstrap starting.
I20260929 03:19:17.248258 401 tablet_bootstrap.cc:492] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b: Bootstrap starting.
I20260929 03:19:17.249656 399 tablet_bootstrap.cc:654] T ea6a1603bb2946a7a80df3f0887b22a2 P ffb3517c69464fbca1d0740b5da4b2e8: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:17.250078 400 tablet_bootstrap.cc:654] T ea6a1603bb2946a7a80df3f0887b22a2 P aab1ea6e786b4d2eb9dfb3ed31c0af06: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:17.250355 401 tablet_bootstrap.cc:654] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:17.250566 399 log.cc:824] T ea6a1603bb2946a7a80df3f0887b22a2 P ffb3517c69464fbca1d0740b5da4b2e8: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:17.251204 400 log.cc:824] T ea6a1603bb2946a7a80df3f0887b22a2 P aab1ea6e786b4d2eb9dfb3ed31c0af06: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:17.251446 401 log.cc:824] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:17.252730 399 tablet_bootstrap.cc:492] T ea6a1603bb2946a7a80df3f0887b22a2 P ffb3517c69464fbca1d0740b5da4b2e8: No bootstrap required, opened a new log
I20260929 03:19:17.252948 399 ts_tablet_manager.cc:1397] T ea6a1603bb2946a7a80df3f0887b22a2 P ffb3517c69464fbca1d0740b5da4b2e8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 03:19:17.253332 400 tablet_bootstrap.cc:492] T ea6a1603bb2946a7a80df3f0887b22a2 P aab1ea6e786b4d2eb9dfb3ed31c0af06: No bootstrap required, opened a new log
I20260929 03:19:17.253535 400 ts_tablet_manager.cc:1397] T ea6a1603bb2946a7a80df3f0887b22a2 P aab1ea6e786b4d2eb9dfb3ed31c0af06: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 03:19:17.253536 401 tablet_bootstrap.cc:492] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b: No bootstrap required, opened a new log
I20260929 03:19:17.253727 401 ts_tablet_manager.cc:1397] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 03:19:17.256605 401 raft_consensus.cc:348] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb3517c69464fbca1d0740b5da4b2e8" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 40779 } } peers { permanent_uuid: "aab1ea6e786b4d2eb9dfb3ed31c0af06" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 43959 } } peers { permanent_uuid: "5dbd39b8dd4d4d9a8e5e00bd10f6bd0b" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 33931 } }
I20260929 03:19:17.256872 401 raft_consensus.cc:374] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:17.256927 401 raft_consensus.cc:729] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b, State: Initialized, Role: FOLLOWER
I20260929 03:19:17.257381 401 consensus_queue.cc:260] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb3517c69464fbca1d0740b5da4b2e8" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 40779 } } peers { permanent_uuid: "aab1ea6e786b4d2eb9dfb3ed31c0af06" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 43959 } } peers { permanent_uuid: "5dbd39b8dd4d4d9a8e5e00bd10f6bd0b" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 33931 } }
I20260929 03:19:17.257385 399 raft_consensus.cc:348] T ea6a1603bb2946a7a80df3f0887b22a2 P ffb3517c69464fbca1d0740b5da4b2e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb3517c69464fbca1d0740b5da4b2e8" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 40779 } } peers { permanent_uuid: "aab1ea6e786b4d2eb9dfb3ed31c0af06" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 43959 } } peers { permanent_uuid: "5dbd39b8dd4d4d9a8e5e00bd10f6bd0b" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 33931 } }
I20260929 03:19:17.257385 400 raft_consensus.cc:348] T ea6a1603bb2946a7a80df3f0887b22a2 P aab1ea6e786b4d2eb9dfb3ed31c0af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb3517c69464fbca1d0740b5da4b2e8" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 40779 } } peers { permanent_uuid: "aab1ea6e786b4d2eb9dfb3ed31c0af06" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 43959 } } peers { permanent_uuid: "5dbd39b8dd4d4d9a8e5e00bd10f6bd0b" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 33931 } }
I20260929 03:19:17.257664 399 raft_consensus.cc:374] T ea6a1603bb2946a7a80df3f0887b22a2 P ffb3517c69464fbca1d0740b5da4b2e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:17.257664 400 raft_consensus.cc:374] T ea6a1603bb2946a7a80df3f0887b22a2 P aab1ea6e786b4d2eb9dfb3ed31c0af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:17.257719 400 raft_consensus.cc:729] T ea6a1603bb2946a7a80df3f0887b22a2 P aab1ea6e786b4d2eb9dfb3ed31c0af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab1ea6e786b4d2eb9dfb3ed31c0af06, State: Initialized, Role: FOLLOWER
I20260929 03:19:17.257720 399 raft_consensus.cc:729] T ea6a1603bb2946a7a80df3f0887b22a2 P ffb3517c69464fbca1d0740b5da4b2e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffb3517c69464fbca1d0740b5da4b2e8, State: Initialized, Role: FOLLOWER
I20260929 03:19:17.258167 399 consensus_queue.cc:260] T ea6a1603bb2946a7a80df3f0887b22a2 P ffb3517c69464fbca1d0740b5da4b2e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb3517c69464fbca1d0740b5da4b2e8" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 40779 } } peers { permanent_uuid: "aab1ea6e786b4d2eb9dfb3ed31c0af06" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 43959 } } peers { permanent_uuid: "5dbd39b8dd4d4d9a8e5e00bd10f6bd0b" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 33931 } }
I20260929 03:19:17.258167 400 consensus_queue.cc:260] T ea6a1603bb2946a7a80df3f0887b22a2 P aab1ea6e786b4d2eb9dfb3ed31c0af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb3517c69464fbca1d0740b5da4b2e8" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 40779 } } peers { permanent_uuid: "aab1ea6e786b4d2eb9dfb3ed31c0af06" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 43959 } } peers { permanent_uuid: "5dbd39b8dd4d4d9a8e5e00bd10f6bd0b" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 33931 } }
I20260929 03:19:17.258451 401 ts_tablet_manager.cc:1428] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260929 03:19:17.258822 32716 heartbeater.cc:503] Master 127.13.238.190:38451 was elected leader, sending a full tablet report...
I20260929 03:19:17.259150 400 ts_tablet_manager.cc:1428] T ea6a1603bb2946a7a80df3f0887b22a2 P aab1ea6e786b4d2eb9dfb3ed31c0af06: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 03:19:17.259150 399 ts_tablet_manager.cc:1428] T ea6a1603bb2946a7a80df3f0887b22a2 P ffb3517c69464fbca1d0740b5da4b2e8: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260929 03:19:17.259503 382 heartbeater.cc:503] Master 127.13.238.190:38451 was elected leader, sending a full tablet report...
I20260929 03:19:17.259675 32582 heartbeater.cc:503] Master 127.13.238.190:38451 was elected leader, sending a full tablet report...
I20260929 03:19:17.392963 405 raft_consensus.cc:482] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:19:17.393178 405 raft_consensus.cc:504] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb3517c69464fbca1d0740b5da4b2e8" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 40779 } } peers { permanent_uuid: "aab1ea6e786b4d2eb9dfb3ed31c0af06" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 43959 } } peers { permanent_uuid: "5dbd39b8dd4d4d9a8e5e00bd10f6bd0b" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 33931 } }
I20260929 03:19:17.394379 405 leader_election.cc:288] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffb3517c69464fbca1d0740b5da4b2e8 (127.13.238.129:40779), aab1ea6e786b4d2eb9dfb3ed31c0af06 (127.13.238.131:43959)
I20260929 03:19:17.399055 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:17.394641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:40779 (local address 127.13.238.130:51857)
0929 03:19:17.394774 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:17.394787 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:17.394893 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:17.395249 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:17.395253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:17.395275 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:17.395413 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:17.395420 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.396363 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.396365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:17.397633 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:17.397645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.398567 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.398571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:17.398689 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:17.398717 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 03:19:17.398802 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 03:19:17.398884 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":1}
I20260929 03:19:17.399091 398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:17.395061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:48481 (local address 127.13.238.131:43959)
0929 03:19:17.395204 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:17.395209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:17.395346 (+ 137us) server_negotiation.cc:407] Connection header received
0929 03:19:17.395440 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:17.395442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:17.395490 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:17.395567 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:17.395921 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.396542 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.397920 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.398575 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.398671 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:17.398706 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:17.398890 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 03:19:17.398963 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:19:17.399091 394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:17.394768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:51857 (local address 127.13.238.129:40779)
0929 03:19:17.394892 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:17.394897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:17.394965 (+ 68us) server_negotiation.cc:407] Connection header received
0929 03:19:17.395021 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:17.395024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:17.395067 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:17.395129 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:17.395525 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.396252 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.397779 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.398394 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.398516 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:17.398573 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:17.398826 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 03:19:17.398969 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:19:17.399063 408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:17.394929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:43959 (local address 127.13.238.130:48481)
0929 03:19:17.395255 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:17.395267 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:17.395361 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:17.395677 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:17.395681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:17.395703 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:17.395815 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:17.395820 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.396647 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.396651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:17.397819 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:17.397826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.398700 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.398703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:17.398787 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:17.398806 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 03:19:17.398866 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 03:19:17.398915 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":138,"threads_started":1}
I20260929 03:19:17.399827 336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea6a1603bb2946a7a80df3f0887b22a2" candidate_uuid: "5dbd39b8dd4d4d9a8e5e00bd10f6bd0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aab1ea6e786b4d2eb9dfb3ed31c0af06" is_pre_election: true
I20260929 03:19:17.399827 32536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea6a1603bb2946a7a80df3f0887b22a2" candidate_uuid: "5dbd39b8dd4d4d9a8e5e00bd10f6bd0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffb3517c69464fbca1d0740b5da4b2e8" is_pre_election: true
I20260929 03:19:17.400177 32536 raft_consensus.cc:2454] T ea6a1603bb2946a7a80df3f0887b22a2 P ffb3517c69464fbca1d0740b5da4b2e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b in term 0.
I20260929 03:19:17.400177 336 raft_consensus.cc:2454] T ea6a1603bb2946a7a80df3f0887b22a2 P aab1ea6e786b4d2eb9dfb3ed31c0af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b in term 0.
I20260929 03:19:17.400658 32606 leader_election.cc:302] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b, ffb3517c69464fbca1d0740b5da4b2e8; no voters:
I20260929 03:19:17.400959 405 raft_consensus.cc:2781] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:19:17.401047 405 raft_consensus.cc:482] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:19:17.401101 405 raft_consensus.cc:3037] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:17.402123 405 raft_consensus.cc:504] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb3517c69464fbca1d0740b5da4b2e8" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 40779 } } peers { permanent_uuid: "aab1ea6e786b4d2eb9dfb3ed31c0af06" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 43959 } } peers { permanent_uuid: "5dbd39b8dd4d4d9a8e5e00bd10f6bd0b" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 33931 } }
I20260929 03:19:17.402472 405 leader_election.cc:288] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b [CANDIDATE]: Term 1 election: Requested vote from peers ffb3517c69464fbca1d0740b5da4b2e8 (127.13.238.129:40779), aab1ea6e786b4d2eb9dfb3ed31c0af06 (127.13.238.131:43959)
I20260929 03:19:17.402841 32536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea6a1603bb2946a7a80df3f0887b22a2" candidate_uuid: "5dbd39b8dd4d4d9a8e5e00bd10f6bd0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffb3517c69464fbca1d0740b5da4b2e8"
I20260929 03:19:17.402846 336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea6a1603bb2946a7a80df3f0887b22a2" candidate_uuid: "5dbd39b8dd4d4d9a8e5e00bd10f6bd0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aab1ea6e786b4d2eb9dfb3ed31c0af06"
I20260929 03:19:17.402968 32536 raft_consensus.cc:3037] T ea6a1603bb2946a7a80df3f0887b22a2 P ffb3517c69464fbca1d0740b5da4b2e8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:17.403019 336 raft_consensus.cc:3037] T ea6a1603bb2946a7a80df3f0887b22a2 P aab1ea6e786b4d2eb9dfb3ed31c0af06 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:17.404266 336 raft_consensus.cc:2454] T ea6a1603bb2946a7a80df3f0887b22a2 P aab1ea6e786b4d2eb9dfb3ed31c0af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b in term 1.
I20260929 03:19:17.404266 32536 raft_consensus.cc:2454] T ea6a1603bb2946a7a80df3f0887b22a2 P ffb3517c69464fbca1d0740b5da4b2e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b in term 1.
I20260929 03:19:17.404577 32606 leader_election.cc:302] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b, aab1ea6e786b4d2eb9dfb3ed31c0af06; no voters:
I20260929 03:19:17.404726 405 raft_consensus.cc:2781] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:17.404995 405 raft_consensus.cc:686] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b [term 1 LEADER]: Becoming Leader. State: Replica: 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b, State: Running, Role: LEADER
I20260929 03:19:17.405346 405 consensus_queue.cc:237] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb3517c69464fbca1d0740b5da4b2e8" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 40779 } } peers { permanent_uuid: "aab1ea6e786b4d2eb9dfb3ed31c0af06" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 43959 } } peers { permanent_uuid: "5dbd39b8dd4d4d9a8e5e00bd10f6bd0b" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 33931 } }
I20260929 03:19:17.408819 32391 catalog_manager.cc:5495] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b (127.13.238.130). New cstate: current_term: 1 leader_uuid: "5dbd39b8dd4d4d9a8e5e00bd10f6bd0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb3517c69464fbca1d0740b5da4b2e8" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 40779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aab1ea6e786b4d2eb9dfb3ed31c0af06" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 43959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dbd39b8dd4d4d9a8e5e00bd10f6bd0b" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 33931 } health_report { overall_health: HEALTHY } } }
W20260929 03:19:17.427238 383 tablet.cc:2369] T ea6a1603bb2946a7a80df3f0887b22a2 P aab1ea6e786b4d2eb9dfb3ed31c0af06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:19:17.428659 397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:17.424661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36406 (local address 127.13.238.130:33931)
0929 03:19:17.424787 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:17.424791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:17.424807 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:19:17.424961 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:17.424964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:17.425027 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:17.425130 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:19:17.425322 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.426103 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.427032 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.427346 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.427450 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:17.427803 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:19:17.428266 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:19:17.428395 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:17.428474 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 03:19:17.428514 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:19:17.434844 336 raft_consensus.cc:1261] T ea6a1603bb2946a7a80df3f0887b22a2 P aab1ea6e786b4d2eb9dfb3ed31c0af06 [term 1 FOLLOWER]: Refusing update from remote peer 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:19:17.434887 32536 raft_consensus.cc:1261] T ea6a1603bb2946a7a80df3f0887b22a2 P ffb3517c69464fbca1d0740b5da4b2e8 [term 1 FOLLOWER]: Refusing update from remote peer 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:19:17.435532 409 consensus_queue.cc:1035] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "aab1ea6e786b4d2eb9dfb3ed31c0af06" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 43959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:19:17.435711 405 consensus_queue.cc:1035] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffb3517c69464fbca1d0740b5da4b2e8" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 40779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:19:17.443571 413 mvcc.cc:205] Tried to move back new op lower bound from 7334510417646006272 to 7334510417535320064. Current Snapshot: MvccSnapshot[applied={T|T < 7334510417646006272}]
I20260929 03:19:17.443712 412 mvcc.cc:205] Tried to move back new op lower bound from 7334510417646006272 to 7334510417535320064. Current Snapshot: MvccSnapshot[applied={T|T < 7334510417646006272}]
I20260929 03:19:17.445483 414 mvcc.cc:205] Tried to move back new op lower bound from 7334510417646006272 to 7334510417535320064. Current Snapshot: MvccSnapshot[applied={T|T < 7334510417646006272}]
I20260929 03:19:17.452349 32390 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:19:17.452512 32390 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:19:17.454713 32390 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fbf9cde9851048dcafde5dd82bcd19f5: Sending DeleteTablet for 3 replicas of tablet ea6a1603bb2946a7a80df3f0887b22a2
I20260929 03:19:17.455470 32650 tablet_service.cc:1514] Processing DeleteTablet for tablet ea6a1603bb2946a7a80df3f0887b22a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:19:17 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:36394
I20260929 03:19:17.455470 32516 tablet_service.cc:1514] Processing DeleteTablet for tablet ea6a1603bb2946a7a80df3f0887b22a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:19:17 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:51150
I20260929 03:19:17.455598 316 tablet_service.cc:1514] Processing DeleteTablet for tablet ea6a1603bb2946a7a80df3f0887b22a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:19:17 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:39920
I20260929 03:19:17.456056 424 tablet_replica.cc:330] T ea6a1603bb2946a7a80df3f0887b22a2 P aab1ea6e786b4d2eb9dfb3ed31c0af06: stopping tablet replica
I20260929 03:19:17.456229 422 tablet_replica.cc:330] T ea6a1603bb2946a7a80df3f0887b22a2 P ffb3517c69464fbca1d0740b5da4b2e8: stopping tablet replica
I20260929 03:19:17.456408 424 raft_consensus.cc:2229] T ea6a1603bb2946a7a80df3f0887b22a2 P aab1ea6e786b4d2eb9dfb3ed31c0af06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:19:17.456525 422 raft_consensus.cc:2229] T ea6a1603bb2946a7a80df3f0887b22a2 P ffb3517c69464fbca1d0740b5da4b2e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:19:17.456751 424 raft_consensus.cc:2258] T ea6a1603bb2946a7a80df3f0887b22a2 P aab1ea6e786b4d2eb9dfb3ed31c0af06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:19:17.456811 422 raft_consensus.cc:2258] T ea6a1603bb2946a7a80df3f0887b22a2 P ffb3517c69464fbca1d0740b5da4b2e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:19:17.456873 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 32453
I20260929 03:19:17.457177 423 tablet_replica.cc:330] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b: stopping tablet replica
I20260929 03:19:17.457569 423 raft_consensus.cc:2229] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:19:17.458473 423 raft_consensus.cc:2258] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:19:17.459349 424 ts_tablet_manager.cc:1905] T ea6a1603bb2946a7a80df3f0887b22a2 P aab1ea6e786b4d2eb9dfb3ed31c0af06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:19:17.459830 423 ts_tablet_manager.cc:1905] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:19:17.462033 424 ts_tablet_manager.cc:1918] T ea6a1603bb2946a7a80df3f0887b22a2 P aab1ea6e786b4d2eb9dfb3ed31c0af06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:19:17.462145 424 log.cc:1196] T ea6a1603bb2946a7a80df3f0887b22a2 P aab1ea6e786b4d2eb9dfb3ed31c0af06: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/ea6a1603bb2946a7a80df3f0887b22a2
I20260929 03:19:17.462456 423 ts_tablet_manager.cc:1918] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:19:17.462500 424 ts_tablet_manager.cc:1939] T ea6a1603bb2946a7a80df3f0887b22a2 P aab1ea6e786b4d2eb9dfb3ed31c0af06: Deleting consensus metadata
I20260929 03:19:17.462548 423 log.cc:1196] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790651713532531-14266-0/minicluster-data/ts-1/wal/wals/ea6a1603bb2946a7a80df3f0887b22a2
I20260929 03:19:17.462824 423 ts_tablet_manager.cc:1939] T ea6a1603bb2946a7a80df3f0887b22a2 P 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b: Deleting consensus metadata
I20260929 03:19:17.463789 32378 catalog_manager.cc:4841] TS aab1ea6e786b4d2eb9dfb3ed31c0af06 (127.13.238.131:43959): tablet ea6a1603bb2946a7a80df3f0887b22a2 (table test-table [id=70cae6c7611342e69a8ed7529278978d]) successfully deleted
I20260929 03:19:17.463816 32375 catalog_manager.cc:4841] TS 5dbd39b8dd4d4d9a8e5e00bd10f6bd0b (127.13.238.130:33931): tablet ea6a1603bb2946a7a80df3f0887b22a2 (table test-table [id=70cae6c7611342e69a8ed7529278978d]) successfully deleted
W20260929 03:19:17.465329 32378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.238.129:40779 (error 108)
I20260929 03:19:17.466104 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 32587
I20260929 03:19:17.466383 395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:17.466049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:40779 (local address 127.0.0.1:51154)
0929 03:19:17.466188 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:17.466295 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.129:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 03:19:17.466800 32378 catalog_manager.cc:4584] TS ffb3517c69464fbca1d0740b5da4b2e8 (127.13.238.129:40779): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea6a1603bb2946a7a80df3f0887b22a2: Network error: Client connection negotiation failed: client connection to 127.13.238.129:40779: connect: Connection refused (error 111)
I20260929 03:19:17.471992 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 32721
I20260929 03:19:17.477097 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 32359
2026-09-29T03:19:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2302 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T03:19:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:17Z Disabled control of system clock
I20260929 03:19:17.505658 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:40251
--webserver_interface=127.13.238.190
--webserver_port=0
--builtin_ntp_servers=127.13.238.148:46017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:40251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:19:17.615206 429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:17.615479 429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:17.615515 429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:17.619596 429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:17.619664 429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:17.619678 429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:17.619693 429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:17.619709 429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:17.624644 429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:46017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:40251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:40251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:17.625773 429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:17.627238 429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:17.635751 434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:17.635735 435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:17.635738 437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:17.636132 429 server_base.cc:1034] running on GCE node
I20260929 03:19:17.636698 429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:17.637801 429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:17.638957 429 hybrid_clock.cc:648] HybridClock initialized: now 1790651957638940 us; error 29 us; skew 500 ppm
I20260929 03:19:17.640795 429 webserver.cc:460] Webserver started at http://127.13.238.190:45729/ using document root <none> and password file <none>
I20260929 03:19:17.641338 429 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:17.641402 429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:17.641609 429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:17.643671 429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "346facb3a82c4b40b276c071fdc83b6c"
format_stamp: "Formatted at 2026-09-29 03:19:17 on dist-test-slave-nbjg"
server_key: "b393a62fe17399f0050b844498bd44c3"
server_key_iv: "152d2faacc55b62215a4945acb1ac4f9"
server_key_version: "encryptionkey@0"
I20260929 03:19:17.644202 429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "346facb3a82c4b40b276c071fdc83b6c"
format_stamp: "Formatted at 2026-09-29 03:19:17 on dist-test-slave-nbjg"
server_key: "b393a62fe17399f0050b844498bd44c3"
server_key_iv: "152d2faacc55b62215a4945acb1ac4f9"
server_key_version: "encryptionkey@0"
I20260929 03:19:17.647909 429 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 03:19:17.650297 443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:17.651582 429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:19:17.651705 429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "346facb3a82c4b40b276c071fdc83b6c"
format_stamp: "Formatted at 2026-09-29 03:19:17 on dist-test-slave-nbjg"
server_key: "b393a62fe17399f0050b844498bd44c3"
server_key_iv: "152d2faacc55b62215a4945acb1ac4f9"
server_key_version: "encryptionkey@0"
I20260929 03:19:17.651800 429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:17.678530 429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:17.679335 429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:17.679518 429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:17.686167 429 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:40251
I20260929 03:19:17.686303 495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:40251 every 8 connection(s)
I20260929 03:19:17.687224 429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:17.690694 496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:17.691563 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 429
I20260929 03:19:17.691670 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:19:17.691970 14266 external_mini_cluster.cc:1428] Setting key 99b98c05cb59b3da2f21ae6eb2976ee9
I20260929 03:19:17.697430 496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 346facb3a82c4b40b276c071fdc83b6c: Bootstrap starting.
I20260929 03:19:17.699887 496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 346facb3a82c4b40b276c071fdc83b6c: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:17.700672 496 log.cc:824] T 00000000000000000000000000000000 P 346facb3a82c4b40b276c071fdc83b6c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:17.702150 499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:17.693567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38324 (local address 127.13.238.190:40251)
0929 03:19:17.694365 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:17.694386 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:17.694435 (+ 49us) server_negotiation.cc:407] Connection header received
0929 03:19:17.695384 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:17.695408 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:17.695864 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:17.696203 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:17.696670 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.697736 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.698714 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.699216 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.699370 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:17.700326 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:17.700331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:17.700347 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:17.700765 (+ 418us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:17.700838 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:17.700868 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:17.700973 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:17.701117 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 03:19:17.701396 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":131,"threads_started":1}
I20260929 03:19:17.702867 496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 346facb3a82c4b40b276c071fdc83b6c: No bootstrap required, opened a new log
I20260929 03:19:17.706687 496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 346facb3a82c4b40b276c071fdc83b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "346facb3a82c4b40b276c071fdc83b6c" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 40251 } }
I20260929 03:19:17.706938 496 raft_consensus.cc:374] T 00000000000000000000000000000000 P 346facb3a82c4b40b276c071fdc83b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:17.707029 496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 346facb3a82c4b40b276c071fdc83b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346facb3a82c4b40b276c071fdc83b6c, State: Initialized, Role: FOLLOWER
I20260929 03:19:17.707466 496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 346facb3a82c4b40b276c071fdc83b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "346facb3a82c4b40b276c071fdc83b6c" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 40251 } }
I20260929 03:19:17.707593 496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 346facb3a82c4b40b276c071fdc83b6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:17.707655 496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 346facb3a82c4b40b276c071fdc83b6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:17.707748 496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 346facb3a82c4b40b276c071fdc83b6c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:17.708709 496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 346facb3a82c4b40b276c071fdc83b6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "346facb3a82c4b40b276c071fdc83b6c" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 40251 } }
I20260929 03:19:17.709074 496 leader_election.cc:302] T 00000000000000000000000000000000 P 346facb3a82c4b40b276c071fdc83b6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 346facb3a82c4b40b276c071fdc83b6c; no voters:
I20260929 03:19:17.709447 496 leader_election.cc:288] T 00000000000000000000000000000000 P 346facb3a82c4b40b276c071fdc83b6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:19:17.709543 501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 346facb3a82c4b40b276c071fdc83b6c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:17.709759 501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 346facb3a82c4b40b276c071fdc83b6c [term 1 LEADER]: Becoming Leader. State: Replica: 346facb3a82c4b40b276c071fdc83b6c, State: Running, Role: LEADER
I20260929 03:19:17.710105 501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 346facb3a82c4b40b276c071fdc83b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "346facb3a82c4b40b276c071fdc83b6c" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 40251 } }
I20260929 03:19:17.710757 496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 346facb3a82c4b40b276c071fdc83b6c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:17.712464 502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 346facb3a82c4b40b276c071fdc83b6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "346facb3a82c4b40b276c071fdc83b6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "346facb3a82c4b40b276c071fdc83b6c" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 40251 } } }
I20260929 03:19:17.712616 502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 346facb3a82c4b40b276c071fdc83b6c [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:17.712862 503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 346facb3a82c4b40b276c071fdc83b6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 346facb3a82c4b40b276c071fdc83b6c. Latest consensus state: current_term: 1 leader_uuid: "346facb3a82c4b40b276c071fdc83b6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "346facb3a82c4b40b276c071fdc83b6c" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 40251 } } }
I20260929 03:19:17.712967 503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 346facb3a82c4b40b276c071fdc83b6c [sys.catalog]: This master's current role is: LEADER
W20260929 03:19:17.715790 517 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 346facb3a82c4b40b276c071fdc83b6c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:19:17.715906 517 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:19:17.716085 516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:17.717067 516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:17.723150 516 catalog_manager.cc:1347] Generated new cluster ID: eeacd2c841e14b8696df4aeb6a34f65b
I20260929 03:19:17.723217 516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:17.747026 516 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:17.748080 516 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:17.755031 516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 346facb3a82c4b40b276c071fdc83b6c: Generated new TSK 0
I20260929 03:19:17.755812 516 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:19:17.763474 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:0
--local_ip_for_outbound_sockets=127.13.238.129
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:40251
--builtin_ntp_servers=127.13.238.148:46017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:19:17.878690 520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:17.878959 520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:17.878997 520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:17.883060 520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:17.883119 520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:17.883175 520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:19:17.887977 520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:46017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:40251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:17.889106 520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:17.890550 520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:17.899842 528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:17.900007 520 server_base.cc:1034] running on GCE node
W20260929 03:19:17.899874 526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:17.899852 525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:17.900573 520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:17.901173 520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:17.902307 520 hybrid_clock.cc:648] HybridClock initialized: now 1790651957902273 us; error 43 us; skew 500 ppm
I20260929 03:19:17.904088 520 webserver.cc:460] Webserver started at http://127.13.238.129:38295/ using document root <none> and password file <none>
I20260929 03:19:17.904651 520 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:17.904726 520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:17.904906 520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:17.907001 520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-0/data/instance:
uuid: "ca185fea77bd4251a80345845868641c"
format_stamp: "Formatted at 2026-09-29 03:19:17 on dist-test-slave-nbjg"
server_key: "e3344c6feb042e2308666313ab980d82"
server_key_iv: "283c87113d3e968222cc9e1d296a95c6"
server_key_version: "encryptionkey@0"
I20260929 03:19:17.907511 520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance:
uuid: "ca185fea77bd4251a80345845868641c"
format_stamp: "Formatted at 2026-09-29 03:19:17 on dist-test-slave-nbjg"
server_key: "e3344c6feb042e2308666313ab980d82"
server_key_iv: "283c87113d3e968222cc9e1d296a95c6"
server_key_version: "encryptionkey@0"
I20260929 03:19:17.911417 520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:19:17.914089 534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:17.915350 520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:19:17.915462 520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "ca185fea77bd4251a80345845868641c"
format_stamp: "Formatted at 2026-09-29 03:19:17 on dist-test-slave-nbjg"
server_key: "e3344c6feb042e2308666313ab980d82"
server_key_iv: "283c87113d3e968222cc9e1d296a95c6"
server_key_version: "encryptionkey@0"
I20260929 03:19:17.915572 520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:17.940981 520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:17.941706 520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:17.941910 520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:17.942545 520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:17.943511 520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:17.943555 520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:17.943594 520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:17.943621 520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:17.951661 520 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:44803
I20260929 03:19:17.951700 647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:44803 every 8 connection(s)
I20260929 03:19:17.952533 520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:19:17.959985 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 520
I20260929 03:19:17.960115 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance
I20260929 03:19:17.960367 14266 external_mini_cluster.cc:1428] Setting key c91e6645c12e0409224c493981b227a8
I20260929 03:19:17.962174 499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:17.954458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:34389 (local address 127.13.238.190:40251)
0929 03:19:17.954581 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:17.954585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:17.955939 (+ 1354us) server_negotiation.cc:407] Connection header received
0929 03:19:17.957280 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:17.957283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:17.957333 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:17.957407 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:17.958309 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.958852 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.960010 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.960290 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.960379 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:17.961644 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:17.961647 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:17.961650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:17.961664 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:17.961689 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:17.961700 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:17.961766 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:17.961931 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 03:19:17.962033 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:19:17.962561 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:0
--local_ip_for_outbound_sockets=127.13.238.130
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:40251
--builtin_ntp_servers=127.13.238.148:46017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:19:17.963109 650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:17.955019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:40251 (local address 127.13.238.129:34389)
0929 03:19:17.955757 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:17.955802 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:17.957076 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:17.957578 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:17.957585 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:17.957669 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:17.958147 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:17.958159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.958962 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.958964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:17.959874 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:17.959885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:17.960441 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:17.960455 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:17.960644 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:17.961338 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:19:17.961340 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:17.961392 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:19:17.961471 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:17.961780 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:17.961790 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:17.961796 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:19:17.961927 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 03:19:17.962186 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":113,"threads_started":1}
I20260929 03:19:17.964269 648 heartbeater.cc:347] Connected to a master server at 127.13.238.190:40251
I20260929 03:19:17.964501 648 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:17.965006 648 heartbeater.cc:511] Master 127.13.238.190:40251 requested a full tablet report, sending...
I20260929 03:19:17.966530 460 ts_manager.cc:194] Registered new tserver with Master: ca185fea77bd4251a80345845868641c (127.13.238.129:44803)
I20260929 03:19:17.967722 460 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.238.129:34389
W20260929 03:19:18.073441 651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:18.073704 651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:18.073741 651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:18.077823 651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:18.077885 651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:18.077941 651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:19:18.082767 651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:46017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:40251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:18.083853 651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:18.085282 651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:18.094863 659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:18.094864 656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:18.094887 657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:18.095125 651 server_base.cc:1034] running on GCE node
I20260929 03:19:18.095559 651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:18.096339 651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:18.097543 651 hybrid_clock.cc:648] HybridClock initialized: now 1790651958097521 us; error 35 us; skew 500 ppm
I20260929 03:19:18.099462 651 webserver.cc:460] Webserver started at http://127.13.238.130:36407/ using document root <none> and password file <none>
I20260929 03:19:18.100013 651 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:18.100072 651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:18.100239 651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:18.102314 651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-1/data/instance:
uuid: "899df479e4634a2db931e59e9286a55d"
format_stamp: "Formatted at 2026-09-29 03:19:18 on dist-test-slave-nbjg"
server_key: "cf9ea062ac70362ade83cf9e6cc9ebc7"
server_key_iv: "06b11ec3182ebbde32153346b5fabdf4"
server_key_version: "encryptionkey@0"
I20260929 03:19:18.102839 651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance:
uuid: "899df479e4634a2db931e59e9286a55d"
format_stamp: "Formatted at 2026-09-29 03:19:18 on dist-test-slave-nbjg"
server_key: "cf9ea062ac70362ade83cf9e6cc9ebc7"
server_key_iv: "06b11ec3182ebbde32153346b5fabdf4"
server_key_version: "encryptionkey@0"
I20260929 03:19:18.106499 651 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:19:18.108867 665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:18.110121 651 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:19:18.110245 651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "899df479e4634a2db931e59e9286a55d"
format_stamp: "Formatted at 2026-09-29 03:19:18 on dist-test-slave-nbjg"
server_key: "cf9ea062ac70362ade83cf9e6cc9ebc7"
server_key_iv: "06b11ec3182ebbde32153346b5fabdf4"
server_key_version: "encryptionkey@0"
I20260929 03:19:18.110342 651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:18.136020 651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:18.136727 651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:18.136911 651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:18.137488 651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:18.138473 651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:18.138520 651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:18.138558 651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:18.138573 651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:18.147177 651 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:39209
I20260929 03:19:18.147202 779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:39209 every 8 connection(s)
I20260929 03:19:18.148087 651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:19:18.148921 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 651
I20260929 03:19:18.149022 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance
I20260929 03:19:18.149279 14266 external_mini_cluster.cc:1428] Setting key e5b48a48865a1c00f4a9e5b446e3c1ed
I20260929 03:19:18.151401 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:0
--local_ip_for_outbound_sockets=127.13.238.131
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:40251
--builtin_ntp_servers=127.13.238.148:46017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:19:18.159492 499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:18.149995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:48971 (local address 127.13.238.190:40251)
0929 03:19:18.150115 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:18.150118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:18.153165 (+ 3047us) server_negotiation.cc:407] Connection header received
0929 03:19:18.154546 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:18.154550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:18.154605 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:18.154678 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:18.155825 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.156503 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.157627 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.157852 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.157939 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:18.159025 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:18.159029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:18.159031 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:18.159045 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:18.159065 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:18.159073 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:18.159129 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:18.159268 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 03:19:18.159332 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:19:18.160383 782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:18.150593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:40251 (local address 127.13.238.130:48971)
0929 03:19:18.153013 (+ 2420us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:18.153055 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:18.154336 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:18.154871 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:18.154881 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:18.154987 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:18.155674 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:18.155695 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.156612 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.156615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:18.157515 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:18.157524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.157989 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.157992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:18.158156 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:18.158756 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:19:18.158758 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:18.158811 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:19:18.158889 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:18.159138 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:18.159147 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:18.159153 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:19:18.159263 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 03:19:18.159543 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2003,"thread_start_us":133,"threads_started":1}
I20260929 03:19:18.161551 780 heartbeater.cc:347] Connected to a master server at 127.13.238.190:40251
I20260929 03:19:18.161806 780 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:18.162415 780 heartbeater.cc:511] Master 127.13.238.190:40251 requested a full tablet report, sending...
I20260929 03:19:18.163604 460 ts_manager.cc:194] Registered new tserver with Master: 899df479e4634a2db931e59e9286a55d (127.13.238.130:39209)
I20260929 03:19:18.164335 460 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.238.130:48971
W20260929 03:19:18.262835 783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:18.263124 783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:18.263162 783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:18.267228 783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:18.267292 783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:18.267349 783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:19:18.272214 783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:46017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:40251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:18.273311 783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:18.274778 783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:18.284399 788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:18.284399 789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:18.284461 791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:18.284664 783 server_base.cc:1034] running on GCE node
I20260929 03:19:18.285128 783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:18.285782 783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:18.286937 783 hybrid_clock.cc:648] HybridClock initialized: now 1790651958286916 us; error 35 us; skew 500 ppm
I20260929 03:19:18.288612 783 webserver.cc:460] Webserver started at http://127.13.238.131:40753/ using document root <none> and password file <none>
I20260929 03:19:18.289173 783 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:18.289235 783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:18.289424 783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:18.291589 783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-2/data/instance:
uuid: "701f021000bb466d82681a57eed843c8"
format_stamp: "Formatted at 2026-09-29 03:19:18 on dist-test-slave-nbjg"
server_key: "7ce7f1928c0bf017f1caabef4757996e"
server_key_iv: "9a9d2510e3f0487cbc49b5da18dff9ed"
server_key_version: "encryptionkey@0"
I20260929 03:19:18.292102 783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance:
uuid: "701f021000bb466d82681a57eed843c8"
format_stamp: "Formatted at 2026-09-29 03:19:18 on dist-test-slave-nbjg"
server_key: "7ce7f1928c0bf017f1caabef4757996e"
server_key_iv: "9a9d2510e3f0487cbc49b5da18dff9ed"
server_key_version: "encryptionkey@0"
I20260929 03:19:18.295643 783 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:19:18.297994 797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:18.299261 783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:19:18.299376 783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "701f021000bb466d82681a57eed843c8"
format_stamp: "Formatted at 2026-09-29 03:19:18 on dist-test-slave-nbjg"
server_key: "7ce7f1928c0bf017f1caabef4757996e"
server_key_iv: "9a9d2510e3f0487cbc49b5da18dff9ed"
server_key_version: "encryptionkey@0"
I20260929 03:19:18.299468 783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:18.321807 783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:18.322611 783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:18.322788 783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:18.323410 783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:18.324380 783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:18.324427 783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:18.324481 783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:18.324507 783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:18.332960 783 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:40921
I20260929 03:19:18.332996 912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:40921 every 8 connection(s)
I20260929 03:19:18.333864 783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:19:18.338449 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 783
I20260929 03:19:18.338557 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance
I20260929 03:19:18.338850 14266 external_mini_cluster.cc:1428] Setting key 56cddbb8a621da3ddbe081c56d7db344
I20260929 03:19:18.343638 499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:18.335750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:46067 (local address 127.13.238.190:40251)
0929 03:19:18.335917 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:18.335921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:18.337027 (+ 1106us) server_negotiation.cc:407] Connection header received
0929 03:19:18.338460 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:18.338464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:18.338525 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:18.338603 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:18.339771 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.340438 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.341773 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.342068 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.342157 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:18.343221 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:18.343224 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:18.343227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:18.343238 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:18.343259 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:18.343267 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:18.343334 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:18.343460 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 03:19:18.343520 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 03:19:18.344485 915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:18.336162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:40251 (local address 127.13.238.131:46067)
0929 03:19:18.336873 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:18.336919 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:18.338179 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:18.338844 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:18.338855 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:18.338974 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:18.339588 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:18.339609 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.340565 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.340569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:18.341645 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:18.341655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.342246 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.342250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:18.342401 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:18.342962 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:19:18.342964 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:18.343012 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:19:18.343081 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:18.343340 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:18.343347 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:18.343352 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:19:18.343449 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 03:19:18.343694 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":114,"threads_started":1}
I20260929 03:19:18.345664 913 heartbeater.cc:347] Connected to a master server at 127.13.238.190:40251
I20260929 03:19:18.345902 913 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:18.346474 913 heartbeater.cc:511] Master 127.13.238.190:40251 requested a full tablet report, sending...
I20260929 03:19:18.347580 460 ts_manager.cc:194] Registered new tserver with Master: 701f021000bb466d82681a57eed843c8 (127.13.238.131:40921)
I20260929 03:19:18.348150 460 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.238.131:46067
I20260929 03:19:18.353013 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:19:18.358827 499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:18.354987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38338 (local address 127.13.238.190:40251)
0929 03:19:18.355121 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:18.355126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:18.355264 (+ 138us) server_negotiation.cc:407] Connection header received
0929 03:19:18.355358 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:18.355361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:18.355401 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:18.355461 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:18.355804 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.356346 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.357184 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.357394 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.357478 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:18.358386 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:18.358389 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:18.358391 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:18.358402 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:18.358421 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:18.358429 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:18.358535 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:18.358640 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 03:19:18.358676 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:19:18.361837 460 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:19:18.364769 460 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:19:18.379379 925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:18.373714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:40921 (local address 127.0.0.1:59544)
0929 03:19:18.374519 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:18.374530 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:18.374604 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:18.375367 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:18.375370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:18.375387 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:18.375475 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:18.375481 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.376955 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.376958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:18.377708 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:18.377717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.378757 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.378760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:18.378889 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:18.378929 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 03:19:18.379034 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 03:19:18.379182 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":78,"threads_started":1}
I20260929 03:19:18.379557 929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:18.374614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59544 (local address 127.13.238.131:40921)
0929 03:19:18.374966 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:18.374976 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:18.374997 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:19:18.375066 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:18.375071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:18.375228 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:18.375631 (+ 403us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:18.375759 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.376846 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.377854 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.378652 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.378759 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:18.378951 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:18.379199 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 03:19:18.379312 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":137,"threads_started":1}
I20260929 03:19:18.379395 927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:18.373377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:44803 (local address 127.0.0.1:52394)
0929 03:19:18.374083 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:18.374141 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:18.374302 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:18.375080 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:18.375086 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:18.375110 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:18.375252 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:18.375259 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.376809 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.376812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:18.377906 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:18.377914 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.378921 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.378925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:18.379012 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:18.379034 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 03:19:18.379082 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 03:19:18.379191 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":26,"spinlock_wait_cycles":5504,"thread_start_us":80,"threads_started":1}
I20260929 03:19:18.379942 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:18.373528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52394 (local address 127.13.238.129:44803)
0929 03:19:18.374593 (+ 1065us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:18.374604 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:18.374629 (+ 25us) server_negotiation.cc:407] Connection header received
0929 03:19:18.374703 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:18.374707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:18.374908 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:18.375408 (+ 500us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:18.375543 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.376698 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.378024 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.378792 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.379462 (+ 670us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:18.379642 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:18.379698 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 03:19:18.379795 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":961,"thread_start_us":108,"threads_started":1}
I20260929 03:19:18.379385 924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:18.373457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:39209 (local address 127.0.0.1:43430)
0929 03:19:18.374119 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:18.374141 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:18.374302 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:18.375220 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:18.375224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:18.375244 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:18.375339 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:18.375346 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.376475 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.376478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:18.377802 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:18.377814 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.379024 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.379027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:18.379109 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:18.379128 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 03:19:18.379186 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 03:19:18.379244 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"spinlock_wait_cycles":2688,"thread_start_us":108,"threads_started":1}
I20260929 03:19:18.380462 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:18.374239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43430 (local address 127.13.238.130:39209)
0929 03:19:18.374569 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:18.374580 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:18.374607 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:19:18.374690 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:18.374696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:18.374911 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:18.375066 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:18.375484 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.376353 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.377955 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.378788 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.378923 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:18.379107 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:18.379454 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 03:19:18.379986 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":123,"threads_started":1}
I20260929 03:19:18.382727 582 tablet_service.cc:1467] Processing CreateTablet for tablet 34680cc929e64c33b3c58f76d756eeb2 (DEFAULT_TABLE table=test-table [id=6a2d127d0b2f4721bfd85c1e1cc28ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:19:18.382776 845 tablet_service.cc:1467] Processing CreateTablet for tablet 34680cc929e64c33b3c58f76d756eeb2 (DEFAULT_TABLE table=test-table [id=6a2d127d0b2f4721bfd85c1e1cc28ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:19:18.382979 713 tablet_service.cc:1467] Processing CreateTablet for tablet 34680cc929e64c33b3c58f76d756eeb2 (DEFAULT_TABLE table=test-table [id=6a2d127d0b2f4721bfd85c1e1cc28ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:19:18.383620 582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34680cc929e64c33b3c58f76d756eeb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:18.383800 713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34680cc929e64c33b3c58f76d756eeb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:18.383800 845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34680cc929e64c33b3c58f76d756eeb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:18.389567 930 tablet_bootstrap.cc:492] T 34680cc929e64c33b3c58f76d756eeb2 P ca185fea77bd4251a80345845868641c: Bootstrap starting.
I20260929 03:19:18.391352 931 tablet_bootstrap.cc:492] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8: Bootstrap starting.
I20260929 03:19:18.391453 932 tablet_bootstrap.cc:492] T 34680cc929e64c33b3c58f76d756eeb2 P 899df479e4634a2db931e59e9286a55d: Bootstrap starting.
I20260929 03:19:18.392076 930 tablet_bootstrap.cc:654] T 34680cc929e64c33b3c58f76d756eeb2 P ca185fea77bd4251a80345845868641c: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:18.392771 930 log.cc:824] T 34680cc929e64c33b3c58f76d756eeb2 P ca185fea77bd4251a80345845868641c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:18.393486 932 tablet_bootstrap.cc:654] T 34680cc929e64c33b3c58f76d756eeb2 P 899df479e4634a2db931e59e9286a55d: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:18.393486 931 tablet_bootstrap.cc:654] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:18.394340 932 log.cc:824] T 34680cc929e64c33b3c58f76d756eeb2 P 899df479e4634a2db931e59e9286a55d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:18.394340 931 log.cc:824] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:18.394565 930 tablet_bootstrap.cc:492] T 34680cc929e64c33b3c58f76d756eeb2 P ca185fea77bd4251a80345845868641c: No bootstrap required, opened a new log
I20260929 03:19:18.394750 930 ts_tablet_manager.cc:1397] T 34680cc929e64c33b3c58f76d756eeb2 P ca185fea77bd4251a80345845868641c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 03:19:18.396274 932 tablet_bootstrap.cc:492] T 34680cc929e64c33b3c58f76d756eeb2 P 899df479e4634a2db931e59e9286a55d: No bootstrap required, opened a new log
I20260929 03:19:18.396274 931 tablet_bootstrap.cc:492] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8: No bootstrap required, opened a new log
I20260929 03:19:18.396457 931 ts_tablet_manager.cc:1397] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260929 03:19:18.396559 932 ts_tablet_manager.cc:1397] T 34680cc929e64c33b3c58f76d756eeb2 P 899df479e4634a2db931e59e9286a55d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260929 03:19:18.398273 930 raft_consensus.cc:348] T 34680cc929e64c33b3c58f76d756eeb2 P ca185fea77bd4251a80345845868641c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca185fea77bd4251a80345845868641c" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 44803 } } peers { permanent_uuid: "899df479e4634a2db931e59e9286a55d" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 39209 } } peers { permanent_uuid: "701f021000bb466d82681a57eed843c8" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40921 } }
I20260929 03:19:18.398494 930 raft_consensus.cc:374] T 34680cc929e64c33b3c58f76d756eeb2 P ca185fea77bd4251a80345845868641c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:18.398551 930 raft_consensus.cc:729] T 34680cc929e64c33b3c58f76d756eeb2 P ca185fea77bd4251a80345845868641c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca185fea77bd4251a80345845868641c, State: Initialized, Role: FOLLOWER
I20260929 03:19:18.398985 930 consensus_queue.cc:260] T 34680cc929e64c33b3c58f76d756eeb2 P ca185fea77bd4251a80345845868641c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca185fea77bd4251a80345845868641c" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 44803 } } peers { permanent_uuid: "899df479e4634a2db931e59e9286a55d" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 39209 } } peers { permanent_uuid: "701f021000bb466d82681a57eed843c8" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40921 } }
I20260929 03:19:18.399611 648 heartbeater.cc:503] Master 127.13.238.190:40251 was elected leader, sending a full tablet report...
I20260929 03:19:18.399977 930 ts_tablet_manager.cc:1428] T 34680cc929e64c33b3c58f76d756eeb2 P ca185fea77bd4251a80345845868641c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 03:19:18.400768 931 raft_consensus.cc:348] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca185fea77bd4251a80345845868641c" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 44803 } } peers { permanent_uuid: "899df479e4634a2db931e59e9286a55d" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 39209 } } peers { permanent_uuid: "701f021000bb466d82681a57eed843c8" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40921 } }
I20260929 03:19:18.400768 932 raft_consensus.cc:348] T 34680cc929e64c33b3c58f76d756eeb2 P 899df479e4634a2db931e59e9286a55d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca185fea77bd4251a80345845868641c" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 44803 } } peers { permanent_uuid: "899df479e4634a2db931e59e9286a55d" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 39209 } } peers { permanent_uuid: "701f021000bb466d82681a57eed843c8" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40921 } }
I20260929 03:19:18.401031 932 raft_consensus.cc:374] T 34680cc929e64c33b3c58f76d756eeb2 P 899df479e4634a2db931e59e9286a55d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:18.401031 931 raft_consensus.cc:374] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:18.401087 931 raft_consensus.cc:729] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 701f021000bb466d82681a57eed843c8, State: Initialized, Role: FOLLOWER
I20260929 03:19:18.401087 932 raft_consensus.cc:729] T 34680cc929e64c33b3c58f76d756eeb2 P 899df479e4634a2db931e59e9286a55d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 899df479e4634a2db931e59e9286a55d, State: Initialized, Role: FOLLOWER
I20260929 03:19:18.401549 931 consensus_queue.cc:260] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca185fea77bd4251a80345845868641c" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 44803 } } peers { permanent_uuid: "899df479e4634a2db931e59e9286a55d" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 39209 } } peers { permanent_uuid: "701f021000bb466d82681a57eed843c8" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40921 } }
I20260929 03:19:18.401549 932 consensus_queue.cc:260] T 34680cc929e64c33b3c58f76d756eeb2 P 899df479e4634a2db931e59e9286a55d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca185fea77bd4251a80345845868641c" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 44803 } } peers { permanent_uuid: "899df479e4634a2db931e59e9286a55d" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 39209 } } peers { permanent_uuid: "701f021000bb466d82681a57eed843c8" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40921 } }
I20260929 03:19:18.402556 931 ts_tablet_manager.cc:1428] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260929 03:19:18.402556 932 ts_tablet_manager.cc:1428] T 34680cc929e64c33b3c58f76d756eeb2 P 899df479e4634a2db931e59e9286a55d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 03:19:18.402910 913 heartbeater.cc:503] Master 127.13.238.190:40251 was elected leader, sending a full tablet report...
I20260929 03:19:18.402905 780 heartbeater.cc:503] Master 127.13.238.190:40251 was elected leader, sending a full tablet report...
W20260929 03:19:18.453666 649 tablet.cc:2369] T 34680cc929e64c33b3c58f76d756eeb2 P ca185fea77bd4251a80345845868641c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:19:18.512118 937 raft_consensus.cc:482] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:19:18.512373 937 raft_consensus.cc:504] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca185fea77bd4251a80345845868641c" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 44803 } } peers { permanent_uuid: "899df479e4634a2db931e59e9286a55d" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 39209 } } peers { permanent_uuid: "701f021000bb466d82681a57eed843c8" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40921 } }
I20260929 03:19:18.513670 937 leader_election.cc:288] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca185fea77bd4251a80345845868641c (127.13.238.129:44803), 899df479e4634a2db931e59e9286a55d (127.13.238.130:39209)
I20260929 03:19:18.517293 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:18.514046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:33529 (local address 127.13.238.129:44803)
0929 03:19:18.514176 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:18.514180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:18.514226 (+ 46us) server_negotiation.cc:407] Connection header received
0929 03:19:18.514396 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:18.514399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:18.514447 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:18.514513 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:18.514802 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.515314 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.516375 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.516762 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.516886 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:18.516940 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:18.517096 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 03:19:18.517172 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:19:18.517293 915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:18.513926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:44803 (local address 127.13.238.131:33529)
0929 03:19:18.514101 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:18.514117 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:18.514250 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:18.514552 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:18.514555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:18.514572 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:18.514679 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:18.514684 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.515429 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.515433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:18.516249 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:18.516259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.516886 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.516890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:18.516988 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:18.517007 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 03:19:18.517072 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 03:19:18.517148 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 03:19:18.517424 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:18.514231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:52669 (local address 127.13.238.130:39209)
0929 03:19:18.514361 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:18.514365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:18.514448 (+ 83us) server_negotiation.cc:407] Connection header received
0929 03:19:18.514589 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:18.514592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:18.514633 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:18.514689 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:18.514988 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.515479 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.516497 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.516907 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.517014 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:18.517063 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:18.517225 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 03:19:18.517294 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:19:18.517458 939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:18.513970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:39209 (local address 127.13.238.131:52669)
0929 03:19:18.514363 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:18.514375 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:18.514483 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:18.514711 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:18.514714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:18.514729 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:18.514880 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:18.514886 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.515593 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.515596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:18.516377 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:18.516386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.517019 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.517022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:18.517116 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:18.517138 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 03:19:18.517203 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 03:19:18.517272 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":115,"threads_started":1}
I20260929 03:19:18.518006 602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34680cc929e64c33b3c58f76d756eeb2" candidate_uuid: "701f021000bb466d82681a57eed843c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca185fea77bd4251a80345845868641c" is_pre_election: true
I20260929 03:19:18.518167 733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34680cc929e64c33b3c58f76d756eeb2" candidate_uuid: "701f021000bb466d82681a57eed843c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "899df479e4634a2db931e59e9286a55d" is_pre_election: true
I20260929 03:19:18.518349 602 raft_consensus.cc:2454] T 34680cc929e64c33b3c58f76d756eeb2 P ca185fea77bd4251a80345845868641c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 701f021000bb466d82681a57eed843c8 in term 0.
I20260929 03:19:18.518517 733 raft_consensus.cc:2454] T 34680cc929e64c33b3c58f76d756eeb2 P 899df479e4634a2db931e59e9286a55d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 701f021000bb466d82681a57eed843c8 in term 0.
I20260929 03:19:18.518896 801 leader_election.cc:302] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 701f021000bb466d82681a57eed843c8, ca185fea77bd4251a80345845868641c; no voters:
I20260929 03:19:18.519162 937 raft_consensus.cc:2781] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:19:18.519248 937 raft_consensus.cc:482] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:19:18.519341 937 raft_consensus.cc:3037] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:18.520424 937 raft_consensus.cc:504] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca185fea77bd4251a80345845868641c" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 44803 } } peers { permanent_uuid: "899df479e4634a2db931e59e9286a55d" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 39209 } } peers { permanent_uuid: "701f021000bb466d82681a57eed843c8" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40921 } }
I20260929 03:19:18.520748 937 leader_election.cc:288] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8 [CANDIDATE]: Term 1 election: Requested vote from peers ca185fea77bd4251a80345845868641c (127.13.238.129:44803), 899df479e4634a2db931e59e9286a55d (127.13.238.130:39209)
I20260929 03:19:18.521126 602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34680cc929e64c33b3c58f76d756eeb2" candidate_uuid: "701f021000bb466d82681a57eed843c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca185fea77bd4251a80345845868641c"
I20260929 03:19:18.521111 733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34680cc929e64c33b3c58f76d756eeb2" candidate_uuid: "701f021000bb466d82681a57eed843c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "899df479e4634a2db931e59e9286a55d"
I20260929 03:19:18.521250 733 raft_consensus.cc:3037] T 34680cc929e64c33b3c58f76d756eeb2 P 899df479e4634a2db931e59e9286a55d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:18.521250 602 raft_consensus.cc:3037] T 34680cc929e64c33b3c58f76d756eeb2 P ca185fea77bd4251a80345845868641c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:18.522522 602 raft_consensus.cc:2454] T 34680cc929e64c33b3c58f76d756eeb2 P ca185fea77bd4251a80345845868641c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 701f021000bb466d82681a57eed843c8 in term 1.
I20260929 03:19:18.522522 733 raft_consensus.cc:2454] T 34680cc929e64c33b3c58f76d756eeb2 P 899df479e4634a2db931e59e9286a55d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 701f021000bb466d82681a57eed843c8 in term 1.
I20260929 03:19:18.522830 798 leader_election.cc:302] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 701f021000bb466d82681a57eed843c8, 899df479e4634a2db931e59e9286a55d; no voters:
I20260929 03:19:18.522989 937 raft_consensus.cc:2781] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:18.523216 937 raft_consensus.cc:686] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8 [term 1 LEADER]: Becoming Leader. State: Replica: 701f021000bb466d82681a57eed843c8, State: Running, Role: LEADER
I20260929 03:19:18.523535 937 consensus_queue.cc:237] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca185fea77bd4251a80345845868641c" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 44803 } } peers { permanent_uuid: "899df479e4634a2db931e59e9286a55d" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 39209 } } peers { permanent_uuid: "701f021000bb466d82681a57eed843c8" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40921 } }
I20260929 03:19:18.526760 455 catalog_manager.cc:5495] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 701f021000bb466d82681a57eed843c8 (127.13.238.131). New cstate: current_term: 1 leader_uuid: "701f021000bb466d82681a57eed843c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca185fea77bd4251a80345845868641c" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 44803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "899df479e4634a2db931e59e9286a55d" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 39209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "701f021000bb466d82681a57eed843c8" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 40921 } health_report { overall_health: HEALTHY } } }
I20260929 03:19:18.570984 929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:18.567767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59554 (local address 127.13.238.131:40921)
0929 03:19:18.567879 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:18.567883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:18.567896 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:19:18.567967 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:18.567969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:18.568024 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:18.568121 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:19:18.568325 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.568825 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.569638 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:18.569844 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:18.569914 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:18.570220 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:19:18.570633 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:19:18.570743 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:18.570819 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 03:19:18.570854 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:19:18.577257 602 raft_consensus.cc:1261] T 34680cc929e64c33b3c58f76d756eeb2 P ca185fea77bd4251a80345845868641c [term 1 FOLLOWER]: Refusing update from remote peer 701f021000bb466d82681a57eed843c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:19:18.577257 733 raft_consensus.cc:1261] T 34680cc929e64c33b3c58f76d756eeb2 P 899df479e4634a2db931e59e9286a55d [term 1 FOLLOWER]: Refusing update from remote peer 701f021000bb466d82681a57eed843c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:19:18.577932 937 consensus_queue.cc:1035] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "899df479e4634a2db931e59e9286a55d" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 39209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:19:18.578115 940 consensus_queue.cc:1035] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca185fea77bd4251a80345845868641c" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 44803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 03:19:18.584723 914 tablet.cc:2369] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:19:18.585526 945 mvcc.cc:205] Tried to move back new op lower bound from 7334510422324850688 to 7334510422115098624. Current Snapshot: MvccSnapshot[applied={T|T < 7334510422324850688}]
I20260929 03:19:18.594121 455 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:19:18.594295 455 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:19:18.596495 455 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 346facb3a82c4b40b276c071fdc83b6c: Sending DeleteTablet for 3 replicas of tablet 34680cc929e64c33b3c58f76d756eeb2
I20260929 03:19:18.597208 845 tablet_service.cc:1514] Processing DeleteTablet for tablet 34680cc929e64c33b3c58f76d756eeb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:19:18 UTC) from {username='slave'} at 127.0.0.1:59544
I20260929 03:19:18.597208 713 tablet_service.cc:1514] Processing DeleteTablet for tablet 34680cc929e64c33b3c58f76d756eeb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:19:18 UTC) from {username='slave'} at 127.0.0.1:43430
I20260929 03:19:18.597227 582 tablet_service.cc:1514] Processing DeleteTablet for tablet 34680cc929e64c33b3c58f76d756eeb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:19:18 UTC) from {username='slave'} at 127.0.0.1:52394
I20260929 03:19:18.597852 957 tablet_replica.cc:330] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8: stopping tablet replica
I20260929 03:19:18.597868 956 tablet_replica.cc:330] T 34680cc929e64c33b3c58f76d756eeb2 P ca185fea77bd4251a80345845868641c: stopping tablet replica
I20260929 03:19:18.598191 956 raft_consensus.cc:2229] T 34680cc929e64c33b3c58f76d756eeb2 P ca185fea77bd4251a80345845868641c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:19:18.598218 957 raft_consensus.cc:2229] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:19:18.598558 956 raft_consensus.cc:2258] T 34680cc929e64c33b3c58f76d756eeb2 P ca185fea77bd4251a80345845868641c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:19:18.598589 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 520
I20260929 03:19:18.599035 955 tablet_replica.cc:330] T 34680cc929e64c33b3c58f76d756eeb2 P 899df479e4634a2db931e59e9286a55d: stopping tablet replica
I20260929 03:19:18.599303 955 raft_consensus.cc:2229] T 34680cc929e64c33b3c58f76d756eeb2 P 899df479e4634a2db931e59e9286a55d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:19:18.599601 955 raft_consensus.cc:2258] T 34680cc929e64c33b3c58f76d756eeb2 P 899df479e4634a2db931e59e9286a55d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:19:18.599732 957 raft_consensus.cc:2258] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:19:18.601289 957 ts_tablet_manager.cc:1905] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:19:18.602612 955 ts_tablet_manager.cc:1905] T 34680cc929e64c33b3c58f76d756eeb2 P 899df479e4634a2db931e59e9286a55d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:19:18.604264 957 ts_tablet_manager.cc:1918] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:19:18.604419 957 log.cc:1196] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/34680cc929e64c33b3c58f76d756eeb2
I20260929 03:19:18.604771 957 ts_tablet_manager.cc:1939] T 34680cc929e64c33b3c58f76d756eeb2 P 701f021000bb466d82681a57eed843c8: Deleting consensus metadata
I20260929 03:19:18.605352 955 ts_tablet_manager.cc:1918] T 34680cc929e64c33b3c58f76d756eeb2 P 899df479e4634a2db931e59e9286a55d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:19:18.605441 955 log.cc:1196] T 34680cc929e64c33b3c58f76d756eeb2 P 899df479e4634a2db931e59e9286a55d: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790651713532531-14266-0/minicluster-data/ts-1/wal/wals/34680cc929e64c33b3c58f76d756eeb2
I20260929 03:19:18.605695 955 ts_tablet_manager.cc:1939] T 34680cc929e64c33b3c58f76d756eeb2 P 899df479e4634a2db931e59e9286a55d: Deleting consensus metadata
I20260929 03:19:18.606241 445 catalog_manager.cc:4841] TS 701f021000bb466d82681a57eed843c8 (127.13.238.131:40921): tablet 34680cc929e64c33b3c58f76d756eeb2 (table test-table [id=6a2d127d0b2f4721bfd85c1e1cc28ff6]) successfully deleted
I20260929 03:19:18.607174 444 catalog_manager.cc:4841] TS 899df479e4634a2db931e59e9286a55d (127.13.238.130:39209): tablet 34680cc929e64c33b3c58f76d756eeb2 (table test-table [id=6a2d127d0b2f4721bfd85c1e1cc28ff6]) successfully deleted
W20260929 03:19:18.607394 447 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.238.129:44803 (error 108)
I20260929 03:19:18.608075 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 651
I20260929 03:19:18.608927 924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:18.608165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:44803 (local address 127.0.0.1:52398)
0929 03:19:18.608777 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:18.608841 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.129:44803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525}
W20260929 03:19:18.609079 447 catalog_manager.cc:4584] TS ca185fea77bd4251a80345845868641c (127.13.238.129:44803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 34680cc929e64c33b3c58f76d756eeb2: Network error: Client connection negotiation failed: client connection to 127.13.238.129:44803: connect: Connection refused (error 111)
I20260929 03:19:18.614732 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 783
I20260929 03:19:18.619771 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 429
2026-09-29T03:19:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1141 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:19:18 dist-test-slave-nbjg krb5kdc[961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:19:18 dist-test-slave-nbjg krb5kdc[961](info): set up 2 sockets
Sep 29 03:19:18 dist-test-slave-nbjg krb5kdc[961](info): commencing operation
krb5kdc: starting...
W20260929 03:19:19.570335 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:19:19 dist-test-slave-nbjg krb5kdc[961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:19:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:19:19.688817 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:35477
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:35477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:19.800051 974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:19.800348 974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:19.800411 974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:19.804505 974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:19.804569 974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:19.804596 974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:19.804616 974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:19.804636 974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:19.809914 974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:35477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:35477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:19.811002 974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:19.812040 974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:19.820413 982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:19.820387 979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:19.820550 974 server_base.cc:1034] running on GCE node
W20260929 03:19:19.820395 980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:19.821192 974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:19.822265 974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:19.823444 974 hybrid_clock.cc:648] HybridClock initialized: now 1790651959823427 us; error 30 us; skew 500 ppm
Sep 29 03:19:19 dist-test-slave-nbjg krb5kdc[961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651959, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:19.826164 974 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:19:19.827098 974 webserver.cc:460] Webserver started at http://127.13.238.190:37109/ using document root <none> and password file <none>
I20260929 03:19:19.827613 974 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:19.827668 974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:19.827836 974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:19.829879 974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "5a848c272cc94aa39a7057de98064e52"
format_stamp: "Formatted at 2026-09-29 03:19:19 on dist-test-slave-nbjg"
server_key: "09390a9d4ffe90cd87796782368692ad"
server_key_iv: "c891c74fcb586d2803ceb7ef5c72f752"
server_key_version: "encryptionkey@0"
I20260929 03:19:19.830420 974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "5a848c272cc94aa39a7057de98064e52"
format_stamp: "Formatted at 2026-09-29 03:19:19 on dist-test-slave-nbjg"
server_key: "09390a9d4ffe90cd87796782368692ad"
server_key_iv: "c891c74fcb586d2803ceb7ef5c72f752"
server_key_version: "encryptionkey@0"
I20260929 03:19:19.834076 974 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:19:19.836547 989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:19.837819 974 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:19:19.837922 974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "5a848c272cc94aa39a7057de98064e52"
format_stamp: "Formatted at 2026-09-29 03:19:19 on dist-test-slave-nbjg"
server_key: "09390a9d4ffe90cd87796782368692ad"
server_key_iv: "c891c74fcb586d2803ceb7ef5c72f752"
server_key_version: "encryptionkey@0"
I20260929 03:19:19.838044 974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:19.856145 974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:19.859210 974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:19.859361 974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:19.866066 974 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:35477
I20260929 03:19:19.866080 1041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:35477 every 8 connection(s)
I20260929 03:19:19.867030 974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:19.869814 1042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:19.875231 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 974
I20260929 03:19:19.875386 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:19:19.875718 14266 external_mini_cluster.cc:1428] Setting key 231320b765d4bae7ad534da81cacb887
I20260929 03:19:19.875864 1042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a848c272cc94aa39a7057de98064e52: Bootstrap starting.
I20260929 03:19:19.878542 1042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a848c272cc94aa39a7057de98064e52: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:19.879384 1042 log.cc:824] T 00000000000000000000000000000000 P 5a848c272cc94aa39a7057de98064e52: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:19.881572 1042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a848c272cc94aa39a7057de98064e52: No bootstrap required, opened a new log
Sep 29 03:19:19 dist-test-slave-nbjg krb5kdc[961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:19:19.886123 1042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a848c272cc94aa39a7057de98064e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a848c272cc94aa39a7057de98064e52" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 35477 } }
I20260929 03:19:19.886443 1042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a848c272cc94aa39a7057de98064e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:19.886507 1042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a848c272cc94aa39a7057de98064e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a848c272cc94aa39a7057de98064e52, State: Initialized, Role: FOLLOWER
I20260929 03:19:19.887012 1042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a848c272cc94aa39a7057de98064e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a848c272cc94aa39a7057de98064e52" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 35477 } }
I20260929 03:19:19.887159 1042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a848c272cc94aa39a7057de98064e52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:19.887229 1042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a848c272cc94aa39a7057de98064e52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:19.887329 1042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a848c272cc94aa39a7057de98064e52 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:19.888429 1042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a848c272cc94aa39a7057de98064e52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a848c272cc94aa39a7057de98064e52" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 35477 } }
I20260929 03:19:19.888782 1042 leader_election.cc:302] T 00000000000000000000000000000000 P 5a848c272cc94aa39a7057de98064e52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a848c272cc94aa39a7057de98064e52; no voters:
I20260929 03:19:19.889115 1042 leader_election.cc:288] T 00000000000000000000000000000000 P 5a848c272cc94aa39a7057de98064e52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:19:19.889235 1047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a848c272cc94aa39a7057de98064e52 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:19.889509 1047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a848c272cc94aa39a7057de98064e52 [term 1 LEADER]: Becoming Leader. State: Replica: 5a848c272cc94aa39a7057de98064e52, State: Running, Role: LEADER
I20260929 03:19:19.889879 1047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a848c272cc94aa39a7057de98064e52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a848c272cc94aa39a7057de98064e52" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 35477 } }
I20260929 03:19:19.890465 1042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a848c272cc94aa39a7057de98064e52 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:19.890937 1045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:19.877183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52692 (local address 127.13.238.190:35477)
0929 03:19:19.877671 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:19.877686 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:19.877722 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:19:19.878431 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:19.878451 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:19.878816 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:19.879102 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:19.880017 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:19.881104 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:19.882303 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:19.882788 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:19.882979 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:19.885688 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:19.885694 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:19.885709 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:19.885738 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:19.887610 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:19.888077 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:19.888082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:19.888087 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:19.888197 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:19.888468 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:19.888470 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:19.888471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:19.888941 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:19.889097 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:19.889439 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 03:19:19.889943 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":152,"threads_started":1}
I20260929 03:19:19.893041 1049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a848c272cc94aa39a7057de98064e52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a848c272cc94aa39a7057de98064e52. Latest consensus state: current_term: 1 leader_uuid: "5a848c272cc94aa39a7057de98064e52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a848c272cc94aa39a7057de98064e52" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 35477 } } }
I20260929 03:19:19.893165 1049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a848c272cc94aa39a7057de98064e52 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:19.893392 1048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a848c272cc94aa39a7057de98064e52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a848c272cc94aa39a7057de98064e52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a848c272cc94aa39a7057de98064e52" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 35477 } } }
I20260929 03:19:19.893479 1048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a848c272cc94aa39a7057de98064e52 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:19.893639 1064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:19.896288 1064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:19.901692 1064 catalog_manager.cc:1347] Generated new cluster ID: 927f239cb3644595914a0920c093477c
I20260929 03:19:19.901808 1064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:19.920363 1064 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:19.921516 1064 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:19.926852 1064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a848c272cc94aa39a7057de98064e52: Generated new TSK 0
I20260929 03:19:19.927558 1064 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
Entry for principal kudu/127.13.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab.
I20260929 03:19:19.954027 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:0
--local_ip_for_outbound_sockets=127.13.238.129
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:35477
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:20.066751 1070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:20.067042 1070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:20.067086 1070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:20.071184 1070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:20.071245 1070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:20.071305 1070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:19:20.076705 1070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.129.keytab
--principal=kudu/127.13.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:35477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:20.077816 1070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:20.078884 1070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:20.088250 1075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:20.088310 1070 server_base.cc:1034] running on GCE node
W20260929 03:19:20.088260 1078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:20.088286 1076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:20.088865 1070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:20.089464 1070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:20.090669 1070 hybrid_clock.cc:648] HybridClock initialized: now 1790651960090635 us; error 40 us; skew 500 ppm
Sep 29 03:19:20 dist-test-slave-nbjg krb5kdc[961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651960, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:20.093484 1070 init.cc:377] Logged in from keytab as kudu/127.13.238.129@KRBTEST.COM (short username kudu)
I20260929 03:19:20.094494 1070 webserver.cc:460] Webserver started at http://127.13.238.129:38609/ using document root <none> and password file <none>
I20260929 03:19:20.095089 1070 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:20.095152 1070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:20.095337 1070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:20.097643 1070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-0/data/instance:
uuid: "0b522b5e6c454eab839efb09ff640317"
format_stamp: "Formatted at 2026-09-29 03:19:20 on dist-test-slave-nbjg"
server_key: "9cc01ba09485a4d99934fd7f46cd10a6"
server_key_iv: "ec853187bd4004576c4954b3db97dfb4"
server_key_version: "encryptionkey@0"
I20260929 03:19:20.098242 1070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance:
uuid: "0b522b5e6c454eab839efb09ff640317"
format_stamp: "Formatted at 2026-09-29 03:19:20 on dist-test-slave-nbjg"
server_key: "9cc01ba09485a4d99934fd7f46cd10a6"
server_key_iv: "ec853187bd4004576c4954b3db97dfb4"
server_key_version: "encryptionkey@0"
I20260929 03:19:20.102176 1070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 03:19:20.104890 1085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:20.106290 1070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 03:19:20.106431 1070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "0b522b5e6c454eab839efb09ff640317"
format_stamp: "Formatted at 2026-09-29 03:19:20 on dist-test-slave-nbjg"
server_key: "9cc01ba09485a4d99934fd7f46cd10a6"
server_key_iv: "ec853187bd4004576c4954b3db97dfb4"
server_key_version: "encryptionkey@0"
I20260929 03:19:20.106534 1070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:20.131947 1070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:20.135216 1070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:20.135380 1070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:20.135993 1070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:20.136906 1070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:20.136957 1070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:20.136999 1070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:20.137012 1070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:20.145560 1070 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:42525
I20260929 03:19:20.145599 1198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:42525 every 8 connection(s)
I20260929 03:19:20.146552 1070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:19:20.150796 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 1070
I20260929 03:19:20.150945 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance
I20260929 03:19:20.151271 14266 external_mini_cluster.cc:1428] Setting key b6ea318abeaf8ef3b31ed7556ce73a8c
Sep 29 03:19:20 dist-test-slave-nbjg krb5kdc[961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651960, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.129@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.238.130@KRBTEST.COM; defaulting to no policy
I20260929 03:19:20.161525 1045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:20.148568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:59273 (local address 127.13.238.190:35477)
0929 03:19:20.148733 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:20.148738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:20.149705 (+ 967us) server_negotiation.cc:407] Connection header received
0929 03:19:20.151488 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:20.151492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:20.151550 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:20.151631 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:20.153111 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.153648 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.155069 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.155316 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.155417 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:20.157909 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:20.157912 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:20.157915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:20.157940 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:20.159354 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:20.159880 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:20.159883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:20.159884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:20.159966 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:20.160290 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:20.160294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:20.160296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:20.160500 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:20.160588 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:20.161185 (+ 597us) server_negotiation.cc:299] Negotiation successful
0929 03:19:20.161329 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.130@KRBTEST.COM" created.
I20260929 03:19:20.162514 1201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:20.148968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:35477 (local address 127.13.238.129:59273)
0929 03:19:20.149524 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:20.149575 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:20.151158 (+ 1583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:20.151845 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:20.151853 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:20.152309 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:20.152951 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:20.152974 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.153778 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.153782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:20.154944 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:20.154954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.155468 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.155472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:20.155642 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:20.156229 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:20.156246 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:20.157730 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:20.159488 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:20.159494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:20.159513 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:20.159780 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:20.160054 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:20.160057 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:20.160059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:20.160203 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:20.160607 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:20.160614 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:20.160936 (+ 322us) client_negotiation.cc:769] Sending connection context
0929 03:19:20.161120 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 03:19:20.161476 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":146,"threads_started":1}
I20260929 03:19:20.163914 1199 heartbeater.cc:347] Connected to a master server at 127.13.238.190:35477
I20260929 03:19:20.164227 1199 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:20.164778 1199 heartbeater.cc:511] Master 127.13.238.190:35477 requested a full tablet report, sending...
I20260929 03:19:20.166648 1006 ts_manager.cc:194] Registered new tserver with Master: 0b522b5e6c454eab839efb09ff640317 (127.13.238.129:42525)
I20260929 03:19:20.168354 1006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.129@KRBTEST.COM'} at 127.13.238.129:59273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
Entry for principal kudu/127.13.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab.
I20260929 03:19:20.179021 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:0
--local_ip_for_outbound_sockets=127.13.238.130
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:35477
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:20.289772 1204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:20.290078 1204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:20.290140 1204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:20.294340 1204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:20.294409 1204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:20.294488 1204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:19:20.299892 1204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.130.keytab
--principal=kudu/127.13.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:35477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:20.300993 1204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:20.302045 1204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:20.311568 1209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:20.311563 1212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:20.311683 1210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:20.311995 1204 server_base.cc:1034] running on GCE node
I20260929 03:19:20.312402 1204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:20.313035 1204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:20.314214 1204 hybrid_clock.cc:648] HybridClock initialized: now 1790651960314171 us; error 35 us; skew 500 ppm
Sep 29 03:19:20 dist-test-slave-nbjg krb5kdc[961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651960, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:20.316859 1204 init.cc:377] Logged in from keytab as kudu/127.13.238.130@KRBTEST.COM (short username kudu)
I20260929 03:19:20.317785 1204 webserver.cc:460] Webserver started at http://127.13.238.130:41335/ using document root <none> and password file <none>
I20260929 03:19:20.318382 1204 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:20.318468 1204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:20.318675 1204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:20.320734 1204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-1/data/instance:
uuid: "0709a8401a1b425482b46cd07281c74f"
format_stamp: "Formatted at 2026-09-29 03:19:20 on dist-test-slave-nbjg"
server_key: "d120463f445670f288481e0f92dfd9af"
server_key_iv: "0a37262ed630fe7cd2a3cadfeef100db"
server_key_version: "encryptionkey@0"
I20260929 03:19:20.321261 1204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance:
uuid: "0709a8401a1b425482b46cd07281c74f"
format_stamp: "Formatted at 2026-09-29 03:19:20 on dist-test-slave-nbjg"
server_key: "d120463f445670f288481e0f92dfd9af"
server_key_iv: "0a37262ed630fe7cd2a3cadfeef100db"
server_key_version: "encryptionkey@0"
I20260929 03:19:20.325022 1204 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260929 03:19:20.327486 1219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:20.328769 1204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 03:19:20.328894 1204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "0709a8401a1b425482b46cd07281c74f"
format_stamp: "Formatted at 2026-09-29 03:19:20 on dist-test-slave-nbjg"
server_key: "d120463f445670f288481e0f92dfd9af"
server_key_iv: "0a37262ed630fe7cd2a3cadfeef100db"
server_key_version: "encryptionkey@0"
I20260929 03:19:20.328996 1204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:20.361012 1204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:20.364261 1204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:20.364461 1204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:20.365072 1204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:20.365934 1204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:20.365983 1204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:20.366037 1204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:20.366065 1204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:20.375000 1204 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:36273
I20260929 03:19:20.375066 1332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:36273 every 8 connection(s)
I20260929 03:19:20.375999 1204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:19:20.385803 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 1204
I20260929 03:19:20.385927 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance
Sep 29 03:19:20 dist-test-slave-nbjg krb5kdc[961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651960, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.130@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:19:20.386227 14266 external_mini_cluster.cc:1428] Setting key fb0a6c156e7c5ad8a2623425b8f5f385
I20260929 03:19:20.389957 1045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:20.378114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:35385 (local address 127.13.238.190:35477)
0929 03:19:20.378314 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:20.378318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:20.379228 (+ 910us) server_negotiation.cc:407] Connection header received
0929 03:19:20.380581 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:20.380584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:20.380635 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:20.380712 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:20.381945 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.382545 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.383745 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.383975 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.384058 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:20.386584 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:20.386589 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:20.386592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:20.386622 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:20.387971 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:20.388459 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:20.388463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:20.388465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:20.388545 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:20.388858 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:20.388862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:20.388864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:20.389084 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:20.389182 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:20.389667 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 03:19:20.389793 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 03:19:20.391201 1335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:20.378532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:35477 (local address 127.13.238.130:35385)
0929 03:19:20.379073 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:20.379119 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:20.380334 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:20.380929 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:20.380938 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:20.381306 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:20.381804 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:20.381818 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.382691 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.382694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:20.383620 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:20.383630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.384135 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.384138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:20.384292 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:20.384859 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:20.384876 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:20.386417 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:20.388085 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:20.388091 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:20.388108 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:20.388364 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:20.388629 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:20.388632 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:20.388634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:20.388773 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:20.389185 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:20.389191 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:20.389446 (+ 255us) client_negotiation.cc:769] Sending connection context
0929 03:19:20.389609 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 03:19:20.389980 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":119,"threads_started":1}
I20260929 03:19:20.392357 1333 heartbeater.cc:347] Connected to a master server at 127.13.238.190:35477
I20260929 03:19:20.392606 1333 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:20.393136 1333 heartbeater.cc:511] Master 127.13.238.190:35477 requested a full tablet report, sending...
I20260929 03:19:20.394326 1006 ts_manager.cc:194] Registered new tserver with Master: 0709a8401a1b425482b46cd07281c74f (127.13.238.130:36273)
I20260929 03:19:20.394887 1006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.130@KRBTEST.COM'} at 127.13.238.130:35385
WARNING: no policy specified for kudu/127.13.238.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
Entry for principal kudu/127.13.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab.
I20260929 03:19:20.413153 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:0
--local_ip_for_outbound_sockets=127.13.238.131
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:35477
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:20.522965 1338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:20.523231 1338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:20.523290 1338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:20.527289 1338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:20.527359 1338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:20.527446 1338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:19:20.533099 1338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.131.keytab
--principal=kudu/127.13.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:35477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:20.534250 1338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:20.535305 1338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:20.544647 1344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:20.544647 1346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:20.544672 1343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:20.544871 1338 server_base.cc:1034] running on GCE node
I20260929 03:19:20.545310 1338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:20.545924 1338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:20.547088 1338 hybrid_clock.cc:648] HybridClock initialized: now 1790651960547056 us; error 49 us; skew 500 ppm
Sep 29 03:19:20 dist-test-slave-nbjg krb5kdc[961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651960, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:20.549647 1338 init.cc:377] Logged in from keytab as kudu/127.13.238.131@KRBTEST.COM (short username kudu)
I20260929 03:19:20.550508 1338 webserver.cc:460] Webserver started at http://127.13.238.131:34919/ using document root <none> and password file <none>
I20260929 03:19:20.551029 1338 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:20.551085 1338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:20.551244 1338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:20.553246 1338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-2/data/instance:
uuid: "144f7ec635e1476ca7791d3449847c99"
format_stamp: "Formatted at 2026-09-29 03:19:20 on dist-test-slave-nbjg"
server_key: "dbc605beeb9cc2a4b539bbb53bc3b7fe"
server_key_iv: "59ce062eb934b1b687f364d37cf4fdea"
server_key_version: "encryptionkey@0"
I20260929 03:19:20.553751 1338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance:
uuid: "144f7ec635e1476ca7791d3449847c99"
format_stamp: "Formatted at 2026-09-29 03:19:20 on dist-test-slave-nbjg"
server_key: "dbc605beeb9cc2a4b539bbb53bc3b7fe"
server_key_iv: "59ce062eb934b1b687f364d37cf4fdea"
server_key_version: "encryptionkey@0"
I20260929 03:19:20.557442 1338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 03:19:20.559810 1353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:20.561028 1338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 03:19:20.561126 1338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "144f7ec635e1476ca7791d3449847c99"
format_stamp: "Formatted at 2026-09-29 03:19:20 on dist-test-slave-nbjg"
server_key: "dbc605beeb9cc2a4b539bbb53bc3b7fe"
server_key_iv: "59ce062eb934b1b687f364d37cf4fdea"
server_key_version: "encryptionkey@0"
I20260929 03:19:20.561229 1338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:20.582535 1338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:20.585403 1338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:20.585579 1338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:20.586190 1338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:20.587069 1338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:20.587121 1338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:20.587167 1338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:20.587195 1338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:20.595919 1338 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:36771
I20260929 03:19:20.595947 1467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:36771 every 8 connection(s)
I20260929 03:19:20.596829 1338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:19:20.599254 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 1338
I20260929 03:19:20.599386 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance
I20260929 03:19:20.599721 14266 external_mini_cluster.cc:1428] Setting key f1ec2f94c1b6e88e9f13919f11e99dd4
Sep 29 03:19:20 dist-test-slave-nbjg krb5kdc[961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651960, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.131@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:19:20.610352 1045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:20.598622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:52719 (local address 127.13.238.190:35477)
0929 03:19:20.598746 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:20.598749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:20.599724 (+ 975us) server_negotiation.cc:407] Connection header received
0929 03:19:20.601324 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:20.601327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:20.601381 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:20.601453 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:20.602757 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.603294 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.604435 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.604663 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.604750 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:20.607069 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:20.607072 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:20.607074 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:20.607100 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:20.608311 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:20.608800 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:20.608804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:20.608806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:20.608896 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:20.609208 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:20.609212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:20.609213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:20.609398 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:20.609483 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:20.610021 (+ 538us) server_negotiation.cc:299] Negotiation successful
0929 03:19:20.610157 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:19:20.611244 1470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:20.599010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:35477 (local address 127.13.238.131:52719)
0929 03:19:20.599539 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:20.599602 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:20.601117 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:20.601662 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:20.601673 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:20.602071 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:20.602624 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:20.602638 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.603407 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.603411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:20.604329 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:20.604341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.604801 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.604805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:20.604970 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:20.605472 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:20.605489 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:20.606900 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:20.608420 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:20.608427 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:20.608444 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:20.608707 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:20.608978 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:20.608981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:20.608983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:20.609123 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:20.609496 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:20.609504 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:20.609790 (+ 286us) client_negotiation.cc:769] Sending connection context
0929 03:19:20.609964 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 03:19:20.610331 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":108,"threads_started":1}
I20260929 03:19:20.612463 1468 heartbeater.cc:347] Connected to a master server at 127.13.238.190:35477
I20260929 03:19:20.612709 1468 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:20.613252 1468 heartbeater.cc:511] Master 127.13.238.190:35477 requested a full tablet report, sending...
I20260929 03:19:20.614310 1006 ts_manager.cc:194] Registered new tserver with Master: 144f7ec635e1476ca7791d3449847c99 (127.13.238.131:36771)
I20260929 03:19:20.614902 1006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.238.131@KRBTEST.COM'} at 127.13.238.131:52719
I20260929 03:19:20.625277 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:19:20.634266 1045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:20.627217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52700 (local address 127.13.238.190:35477)
0929 03:19:20.627333 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:20.627337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:20.627451 (+ 114us) server_negotiation.cc:407] Connection header received
0929 03:19:20.627567 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:20.627569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:20.627607 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:20.627672 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:20.628387 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.628933 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.629714 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.629932 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.630005 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:20.631393 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:20.631396 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:20.631398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:20.631422 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:20.632727 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:20.633159 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:20.633163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:20.633165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:20.633248 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:20.633526 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:20.633528 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:20.633530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:20.633736 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:20.633833 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:20.634024 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 03:19:20.634118 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:19:20.636934 1006 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:19:20.640027 1006 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:19:20.653815 1479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:20.648744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:42525 (local address 127.0.0.1:41578)
0929 03:19:20.649111 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:20.649159 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:20.649296 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:20.649858 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:20.649864 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:20.649889 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:20.650031 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:20.650039 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.651272 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.651275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:20.652180 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:20.652190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.653262 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.653266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:20.653400 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:20.653439 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 03:19:20.653534 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 03:19:20.653666 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260929 03:19:20.653820 1480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:20.648884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41578 (local address 127.13.238.129:42525)
0929 03:19:20.649162 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:20.649180 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:20.649247 (+ 67us) server_negotiation.cc:407] Connection header received
0929 03:19:20.649452 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:20.649459 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:20.649690 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:20.649862 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:20.650173 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.651132 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.652313 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.653158 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.653265 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:20.653406 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:20.653579 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 03:19:20.653675 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":86,"threads_started":1}
I20260929 03:19:20.655509 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:20.649854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:36771 (local address 127.0.0.1:49944)
0929 03:19:20.650349 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:20.650362 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:20.650447 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:20.651078 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:20.651081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:20.651101 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:20.651214 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:20.651221 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.652676 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.652680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:20.654076 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:20.654087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.655108 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.655111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:20.655203 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:20.655223 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 03:19:20.655299 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 03:19:20.655371 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":62,"thread_start_us":96,"threads_started":1}
I20260929 03:19:20.655557 1484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:20.650321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49944 (local address 127.13.238.131:36771)
0929 03:19:20.650696 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:20.650706 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:20.650726 (+ 20us) server_negotiation.cc:407] Connection header received
0929 03:19:20.650789 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:20.650794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:20.650940 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:20.651068 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:20.651338 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.652523 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.654394 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.654980 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.655112 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:20.655235 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:20.655326 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 03:19:20.655419 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":144,"threads_started":1}
I20260929 03:19:20.656323 1482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:20.649718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44466 (local address 127.13.238.130:36273)
0929 03:19:20.650160 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:20.650172 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:20.650439 (+ 267us) server_negotiation.cc:407] Connection header received
0929 03:19:20.650634 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:20.650640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:20.650844 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:20.650962 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:20.651395 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.652523 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.654222 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.655026 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.655138 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:20.655259 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:20.656098 (+ 839us) server_negotiation.cc:299] Negotiation successful
0929 03:19:20.656195 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":114,"threads_started":1}
I20260929 03:19:20.656679 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:20.649565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:36273 (local address 127.0.0.1:44466)
0929 03:19:20.650349 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:20.650362 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:20.650495 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:20.651155 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:20.651158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:20.651176 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:20.651278 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:20.651284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.652676 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.652680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:20.654076 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:20.654087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.655315 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.655319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:20.655409 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:20.655430 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 03:19:20.656460 (+ 1030us) client_negotiation.cc:240] Negotiation successful
0929 03:19:20.656545 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":134,"thread_start_us":92,"threads_started":1}
I20260929 03:19:20.657281 1133 tablet_service.cc:1467] Processing CreateTablet for tablet 7021e7a053464d66aeec426108b34269 (DEFAULT_TABLE table=test-table [id=dfae68b44d4a45b4b2593fceef7a567c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:19:20.658506 1133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7021e7a053464d66aeec426108b34269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:20.658212 1401 tablet_service.cc:1467] Processing CreateTablet for tablet 7021e7a053464d66aeec426108b34269 (DEFAULT_TABLE table=test-table [id=dfae68b44d4a45b4b2593fceef7a567c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:19:20.659183 1401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7021e7a053464d66aeec426108b34269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:20.660995 1267 tablet_service.cc:1467] Processing CreateTablet for tablet 7021e7a053464d66aeec426108b34269 (DEFAULT_TABLE table=test-table [id=dfae68b44d4a45b4b2593fceef7a567c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:19:20.661986 1267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7021e7a053464d66aeec426108b34269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:20.665166 1485 tablet_bootstrap.cc:492] T 7021e7a053464d66aeec426108b34269 P 0b522b5e6c454eab839efb09ff640317: Bootstrap starting.
I20260929 03:19:20.666443 1486 tablet_bootstrap.cc:492] T 7021e7a053464d66aeec426108b34269 P 144f7ec635e1476ca7791d3449847c99: Bootstrap starting.
I20260929 03:19:20.667492 1485 tablet_bootstrap.cc:654] T 7021e7a053464d66aeec426108b34269 P 0b522b5e6c454eab839efb09ff640317: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:20.668362 1485 log.cc:824] T 7021e7a053464d66aeec426108b34269 P 0b522b5e6c454eab839efb09ff640317: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:20.668622 1487 tablet_bootstrap.cc:492] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f: Bootstrap starting.
I20260929 03:19:20.668737 1486 tablet_bootstrap.cc:654] T 7021e7a053464d66aeec426108b34269 P 144f7ec635e1476ca7791d3449847c99: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:20.670064 1486 log.cc:824] T 7021e7a053464d66aeec426108b34269 P 144f7ec635e1476ca7791d3449847c99: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:20.670681 1487 tablet_bootstrap.cc:654] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:20.670818 1485 tablet_bootstrap.cc:492] T 7021e7a053464d66aeec426108b34269 P 0b522b5e6c454eab839efb09ff640317: No bootstrap required, opened a new log
I20260929 03:19:20.671029 1485 ts_tablet_manager.cc:1397] T 7021e7a053464d66aeec426108b34269 P 0b522b5e6c454eab839efb09ff640317: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 03:19:20.671872 1486 tablet_bootstrap.cc:492] T 7021e7a053464d66aeec426108b34269 P 144f7ec635e1476ca7791d3449847c99: No bootstrap required, opened a new log
I20260929 03:19:20.671878 1487 log.cc:824] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:20.672042 1486 ts_tablet_manager.cc:1397] T 7021e7a053464d66aeec426108b34269 P 144f7ec635e1476ca7791d3449847c99: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260929 03:19:20.673743 1487 tablet_bootstrap.cc:492] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f: No bootstrap required, opened a new log
I20260929 03:19:20.673940 1487 ts_tablet_manager.cc:1397] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 03:19:20.674973 1486 raft_consensus.cc:348] T 7021e7a053464d66aeec426108b34269 P 144f7ec635e1476ca7791d3449847c99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b522b5e6c454eab839efb09ff640317" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42525 } } peers { permanent_uuid: "0709a8401a1b425482b46cd07281c74f" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 36273 } } peers { permanent_uuid: "144f7ec635e1476ca7791d3449847c99" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36771 } }
I20260929 03:19:20.675184 1486 raft_consensus.cc:374] T 7021e7a053464d66aeec426108b34269 P 144f7ec635e1476ca7791d3449847c99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:20.675251 1486 raft_consensus.cc:729] T 7021e7a053464d66aeec426108b34269 P 144f7ec635e1476ca7791d3449847c99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 144f7ec635e1476ca7791d3449847c99, State: Initialized, Role: FOLLOWER
I20260929 03:19:20.675208 1485 raft_consensus.cc:348] T 7021e7a053464d66aeec426108b34269 P 0b522b5e6c454eab839efb09ff640317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b522b5e6c454eab839efb09ff640317" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42525 } } peers { permanent_uuid: "0709a8401a1b425482b46cd07281c74f" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 36273 } } peers { permanent_uuid: "144f7ec635e1476ca7791d3449847c99" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36771 } }
I20260929 03:19:20.675472 1485 raft_consensus.cc:374] T 7021e7a053464d66aeec426108b34269 P 0b522b5e6c454eab839efb09ff640317 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:20.675572 1485 raft_consensus.cc:729] T 7021e7a053464d66aeec426108b34269 P 0b522b5e6c454eab839efb09ff640317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b522b5e6c454eab839efb09ff640317, State: Initialized, Role: FOLLOWER
I20260929 03:19:20.675742 1486 consensus_queue.cc:260] T 7021e7a053464d66aeec426108b34269 P 144f7ec635e1476ca7791d3449847c99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b522b5e6c454eab839efb09ff640317" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42525 } } peers { permanent_uuid: "0709a8401a1b425482b46cd07281c74f" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 36273 } } peers { permanent_uuid: "144f7ec635e1476ca7791d3449847c99" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36771 } }
I20260929 03:19:20.676013 1485 consensus_queue.cc:260] T 7021e7a053464d66aeec426108b34269 P 0b522b5e6c454eab839efb09ff640317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b522b5e6c454eab839efb09ff640317" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42525 } } peers { permanent_uuid: "0709a8401a1b425482b46cd07281c74f" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 36273 } } peers { permanent_uuid: "144f7ec635e1476ca7791d3449847c99" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36771 } }
I20260929 03:19:20.676410 1468 heartbeater.cc:503] Master 127.13.238.190:35477 was elected leader, sending a full tablet report...
I20260929 03:19:20.677032 1485 ts_tablet_manager.cc:1428] T 7021e7a053464d66aeec426108b34269 P 0b522b5e6c454eab839efb09ff640317: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 03:19:20.677143 1199 heartbeater.cc:503] Master 127.13.238.190:35477 was elected leader, sending a full tablet report...
I20260929 03:19:20.677263 1486 ts_tablet_manager.cc:1428] T 7021e7a053464d66aeec426108b34269 P 144f7ec635e1476ca7791d3449847c99: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260929 03:19:20.677603 1487 raft_consensus.cc:348] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b522b5e6c454eab839efb09ff640317" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42525 } } peers { permanent_uuid: "0709a8401a1b425482b46cd07281c74f" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 36273 } } peers { permanent_uuid: "144f7ec635e1476ca7791d3449847c99" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36771 } }
I20260929 03:19:20.677771 1487 raft_consensus.cc:374] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:20.677799 1487 raft_consensus.cc:729] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0709a8401a1b425482b46cd07281c74f, State: Initialized, Role: FOLLOWER
I20260929 03:19:20.678160 1487 consensus_queue.cc:260] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b522b5e6c454eab839efb09ff640317" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42525 } } peers { permanent_uuid: "0709a8401a1b425482b46cd07281c74f" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 36273 } } peers { permanent_uuid: "144f7ec635e1476ca7791d3449847c99" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36771 } }
I20260929 03:19:20.679019 1487 ts_tablet_manager.cc:1428] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 03:19:20.680521 1333 heartbeater.cc:503] Master 127.13.238.190:35477 was elected leader, sending a full tablet report...
I20260929 03:19:20.805853 1493 raft_consensus.cc:482] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:19:20.806161 1493 raft_consensus.cc:504] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b522b5e6c454eab839efb09ff640317" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42525 } } peers { permanent_uuid: "0709a8401a1b425482b46cd07281c74f" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 36273 } } peers { permanent_uuid: "144f7ec635e1476ca7791d3449847c99" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36771 } }
I20260929 03:19:20.807359 1493 leader_election.cc:288] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b522b5e6c454eab839efb09ff640317 (127.13.238.129:42525), 144f7ec635e1476ca7791d3449847c99 (127.13.238.131:36771)
I20260929 03:19:20.810942 1480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:20.807788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:33443 (local address 127.13.238.129:42525)
0929 03:19:20.807916 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:20.807920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:20.807936 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:19:20.807998 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:20.808001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:20.808057 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:20.808120 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:20.808395 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.808905 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.809939 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.810406 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.810512 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:20.810566 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:20.810747 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 03:19:20.810814 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:19:20.810942 1335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:20.807658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:42525 (local address 127.13.238.130:33443)
0929 03:19:20.807793 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:20.807807 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:20.807915 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:20.808148 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:20.808151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:20.808171 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:20.808285 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:20.808291 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.809012 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.809015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:20.809816 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:20.809825 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.810512 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.810515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:20.810634 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:20.810654 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 03:19:20.810724 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 03:19:20.810798 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 03:19:20.811645 1484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:20.808071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:49691 (local address 127.13.238.131:36771)
0929 03:19:20.808192 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:20.808196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:20.808404 (+ 208us) server_negotiation.cc:407] Connection header received
0929 03:19:20.808524 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:20.808527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:20.808574 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:20.808637 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:20.808894 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.809420 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.810531 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.811008 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.811127 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:20.811173 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:20.811472 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 03:19:20.811542 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:19:20.811652 1494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:20.807938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:36771 (local address 127.13.238.130:49691)
0929 03:19:20.808316 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:20.808327 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:20.808421 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:20.808659 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:20.808661 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:20.808677 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:20.808797 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:20.808803 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.809530 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.809533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:20.810413 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:20.810422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.811228 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.811253 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:20.811353 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:20.811375 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 03:19:20.811441 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 03:19:20.811506 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":133,"threads_started":1}
I20260929 03:19:20.811605 1153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7021e7a053464d66aeec426108b34269" candidate_uuid: "0709a8401a1b425482b46cd07281c74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b522b5e6c454eab839efb09ff640317" is_pre_election: true
I20260929 03:19:20.811920 1153 raft_consensus.cc:2454] T 7021e7a053464d66aeec426108b34269 P 0b522b5e6c454eab839efb09ff640317 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0709a8401a1b425482b46cd07281c74f in term 0.
I20260929 03:19:20.812285 1421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7021e7a053464d66aeec426108b34269" candidate_uuid: "0709a8401a1b425482b46cd07281c74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "144f7ec635e1476ca7791d3449847c99" is_pre_election: true
I20260929 03:19:20.812438 1222 leader_election.cc:302] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0709a8401a1b425482b46cd07281c74f, 0b522b5e6c454eab839efb09ff640317; no voters:
I20260929 03:19:20.812585 1421 raft_consensus.cc:2454] T 7021e7a053464d66aeec426108b34269 P 144f7ec635e1476ca7791d3449847c99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0709a8401a1b425482b46cd07281c74f in term 0.
I20260929 03:19:20.812772 1493 raft_consensus.cc:2781] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:19:20.812863 1493 raft_consensus.cc:482] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:19:20.812943 1493 raft_consensus.cc:3037] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:20.813997 1493 raft_consensus.cc:504] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b522b5e6c454eab839efb09ff640317" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42525 } } peers { permanent_uuid: "0709a8401a1b425482b46cd07281c74f" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 36273 } } peers { permanent_uuid: "144f7ec635e1476ca7791d3449847c99" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36771 } }
I20260929 03:19:20.814396 1493 leader_election.cc:288] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f [CANDIDATE]: Term 1 election: Requested vote from peers 0b522b5e6c454eab839efb09ff640317 (127.13.238.129:42525), 144f7ec635e1476ca7791d3449847c99 (127.13.238.131:36771)
I20260929 03:19:20.814771 1153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7021e7a053464d66aeec426108b34269" candidate_uuid: "0709a8401a1b425482b46cd07281c74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b522b5e6c454eab839efb09ff640317"
I20260929 03:19:20.814808 1421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7021e7a053464d66aeec426108b34269" candidate_uuid: "0709a8401a1b425482b46cd07281c74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "144f7ec635e1476ca7791d3449847c99"
I20260929 03:19:20.814896 1153 raft_consensus.cc:3037] T 7021e7a053464d66aeec426108b34269 P 0b522b5e6c454eab839efb09ff640317 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:20.814913 1421 raft_consensus.cc:3037] T 7021e7a053464d66aeec426108b34269 P 144f7ec635e1476ca7791d3449847c99 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:20.816002 1421 raft_consensus.cc:2454] T 7021e7a053464d66aeec426108b34269 P 144f7ec635e1476ca7791d3449847c99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0709a8401a1b425482b46cd07281c74f in term 1.
I20260929 03:19:20.816049 1153 raft_consensus.cc:2454] T 7021e7a053464d66aeec426108b34269 P 0b522b5e6c454eab839efb09ff640317 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0709a8401a1b425482b46cd07281c74f in term 1.
I20260929 03:19:20.816314 1222 leader_election.cc:302] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0709a8401a1b425482b46cd07281c74f, 144f7ec635e1476ca7791d3449847c99; no voters:
I20260929 03:19:20.816530 1493 raft_consensus.cc:2781] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:20.816749 1493 raft_consensus.cc:686] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f [term 1 LEADER]: Becoming Leader. State: Replica: 0709a8401a1b425482b46cd07281c74f, State: Running, Role: LEADER
I20260929 03:19:20.817127 1493 consensus_queue.cc:237] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b522b5e6c454eab839efb09ff640317" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42525 } } peers { permanent_uuid: "0709a8401a1b425482b46cd07281c74f" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 36273 } } peers { permanent_uuid: "144f7ec635e1476ca7791d3449847c99" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36771 } }
I20260929 03:19:20.820387 1006 catalog_manager.cc:5495] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0709a8401a1b425482b46cd07281c74f (127.13.238.130). New cstate: current_term: 1 leader_uuid: "0709a8401a1b425482b46cd07281c74f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b522b5e6c454eab839efb09ff640317" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0709a8401a1b425482b46cd07281c74f" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 36273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "144f7ec635e1476ca7791d3449847c99" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36771 } health_report { overall_health: UNKNOWN } } }
I20260929 03:19:20.844774 1482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:20.841319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44482 (local address 127.13.238.130:36273)
0929 03:19:20.841436 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:20.841439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:20.841454 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:19:20.841643 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:20.841646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:20.841707 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:20.841795 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:19:20.841964 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.842508 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.843382 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:20.843613 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:20.843693 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:20.843935 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:19:20.844369 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:19:20.844462 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:20.844556 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 03:19:20.844609 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260929 03:19:20.847900 1469 tablet.cc:2369] T 7021e7a053464d66aeec426108b34269 P 144f7ec635e1476ca7791d3449847c99: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:19:20.850986 1153 raft_consensus.cc:1261] T 7021e7a053464d66aeec426108b34269 P 0b522b5e6c454eab839efb09ff640317 [term 1 FOLLOWER]: Refusing update from remote peer 0709a8401a1b425482b46cd07281c74f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:19:20.850982 1421 raft_consensus.cc:1261] T 7021e7a053464d66aeec426108b34269 P 144f7ec635e1476ca7791d3449847c99 [term 1 FOLLOWER]: Refusing update from remote peer 0709a8401a1b425482b46cd07281c74f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:19:20.851758 1495 consensus_queue.cc:1035] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b522b5e6c454eab839efb09ff640317" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:19:20.851981 1493 consensus_queue.cc:1035] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f [LEADER]: Connected to new peer: Peer: permanent_uuid: "144f7ec635e1476ca7791d3449847c99" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:19:20.860172 1499 mvcc.cc:205] Tried to move back new op lower bound from 7334510431637925888 to 7334510431509774336. Current Snapshot: MvccSnapshot[applied={T|T < 7334510431637925888}]
I20260929 03:19:20.862015 1500 mvcc.cc:205] Tried to move back new op lower bound from 7334510431637925888 to 7334510431509774336. Current Snapshot: MvccSnapshot[applied={T|T < 7334510431637925888}]
I20260929 03:19:20.868692 1006 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:19:20.868865 1006 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:19:20.871423 1006 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a848c272cc94aa39a7057de98064e52: Sending DeleteTablet for 3 replicas of tablet 7021e7a053464d66aeec426108b34269
I20260929 03:19:20.872380 1133 tablet_service.cc:1514] Processing DeleteTablet for tablet 7021e7a053464d66aeec426108b34269 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:19:20 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:41578
I20260929 03:19:20.872738 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 1070
I20260929 03:19:20.872864 1267 tablet_service.cc:1514] Processing DeleteTablet for tablet 7021e7a053464d66aeec426108b34269 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:19:20 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:44466
I20260929 03:19:20.874406 1401 tablet_service.cc:1514] Processing DeleteTablet for tablet 7021e7a053464d66aeec426108b34269 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:19:20 UTC) from {username='kudu', principal='kudu/127.13.238.190@KRBTEST.COM'} at 127.0.0.1:49944
I20260929 03:19:20.874838 1510 tablet_replica.cc:330] T 7021e7a053464d66aeec426108b34269 P 144f7ec635e1476ca7791d3449847c99: stopping tablet replica
I20260929 03:19:20.875216 1510 raft_consensus.cc:2229] T 7021e7a053464d66aeec426108b34269 P 144f7ec635e1476ca7791d3449847c99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:19:20.875531 1510 raft_consensus.cc:2258] T 7021e7a053464d66aeec426108b34269 P 144f7ec635e1476ca7791d3449847c99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:19:20.877177 1510 ts_tablet_manager.cc:1905] T 7021e7a053464d66aeec426108b34269 P 144f7ec635e1476ca7791d3449847c99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:19:20.880019 1510 ts_tablet_manager.cc:1918] T 7021e7a053464d66aeec426108b34269 P 144f7ec635e1476ca7791d3449847c99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:19:20.880129 1510 log.cc:1196] T 7021e7a053464d66aeec426108b34269 P 144f7ec635e1476ca7791d3449847c99: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/7021e7a053464d66aeec426108b34269
I20260929 03:19:20.880484 1510 ts_tablet_manager.cc:1939] T 7021e7a053464d66aeec426108b34269 P 144f7ec635e1476ca7791d3449847c99: Deleting consensus metadata
I20260929 03:19:20.881757 992 catalog_manager.cc:4841] TS 144f7ec635e1476ca7791d3449847c99 (127.13.238.131:36771): tablet 7021e7a053464d66aeec426108b34269 (table test-table [id=dfae68b44d4a45b4b2593fceef7a567c]) successfully deleted
W20260929 03:19:20.885398 992 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.238.129:42525 (error 108)
I20260929 03:19:20.885573 1509 tablet_replica.cc:330] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f: stopping tablet replica
I20260929 03:19:20.885913 1509 raft_consensus.cc:2229] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:19:20.886411 1509 raft_consensus.cc:2258] T 7021e7a053464d66aeec426108b34269 P 0709a8401a1b425482b46cd07281c74f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:19:20.886451 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:20.886141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:42525 (local address 127.0.0.1:41590)
0929 03:19:20.886292 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:20.886364 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.129:42525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 03:19:20.886659 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 1204
W20260929 03:19:20.886665 992 catalog_manager.cc:4584] TS 0b522b5e6c454eab839efb09ff640317 (127.13.238.129:42525): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7021e7a053464d66aeec426108b34269: Network error: Client connection negotiation failed: client connection to 127.13.238.129:42525: connect: Connection refused (error 111)
I20260929 03:19:20.892408 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 1338
I20260929 03:19:20.893173 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:20.892602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:36273 (local address 127.0.0.1:44484)
0929 03:19:20.893035 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:20.893087 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.130:36273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352}
I20260929 03:19:20.897490 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 974
2026-09-29T03:19:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2278 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T03:19:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:20Z Disabled control of system clock
I20260929 03:19:20.926468 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:40681
--webserver_interface=127.13.238.190
--webserver_port=0
--builtin_ntp_servers=127.13.238.148:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:40681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:19:21.039088 1516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:21.039372 1516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:21.039417 1516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:21.043596 1516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:21.043668 1516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:21.043682 1516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:21.043699 1516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:21.043712 1516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:21.048676 1516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:40681
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:40681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:21.049827 1516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:21.051337 1516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:21.059794 1524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:21.059794 1521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:21.059850 1516 server_base.cc:1034] running on GCE node
W20260929 03:19:21.059788 1522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:21.060600 1516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:21.061800 1516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:21.062971 1516 hybrid_clock.cc:648] HybridClock initialized: now 1790651961062950 us; error 39 us; skew 500 ppm
I20260929 03:19:21.064895 1516 webserver.cc:460] Webserver started at http://127.13.238.190:34125/ using document root <none> and password file <none>
I20260929 03:19:21.065483 1516 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:21.065552 1516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:21.065757 1516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:21.067922 1516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "ebcb2fce6613442bb4891c277000fdf5"
format_stamp: "Formatted at 2026-09-29 03:19:21 on dist-test-slave-nbjg"
server_key: "18842025a0724148d75cd037418a0f8b"
server_key_iv: "4a52003d2ed324a2d9aba1d7a28cb311"
server_key_version: "encryptionkey@0"
I20260929 03:19:21.068464 1516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "ebcb2fce6613442bb4891c277000fdf5"
format_stamp: "Formatted at 2026-09-29 03:19:21 on dist-test-slave-nbjg"
server_key: "18842025a0724148d75cd037418a0f8b"
server_key_iv: "4a52003d2ed324a2d9aba1d7a28cb311"
server_key_version: "encryptionkey@0"
I20260929 03:19:21.072158 1516 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 03:19:21.074577 1530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:21.075824 1516 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:19:21.075941 1516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "ebcb2fce6613442bb4891c277000fdf5"
format_stamp: "Formatted at 2026-09-29 03:19:21 on dist-test-slave-nbjg"
server_key: "18842025a0724148d75cd037418a0f8b"
server_key_iv: "4a52003d2ed324a2d9aba1d7a28cb311"
server_key_version: "encryptionkey@0"
I20260929 03:19:21.076053 1516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:21.098188 1516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:21.098994 1516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:21.099169 1516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:21.105997 1516 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:40681
I20260929 03:19:21.106051 1583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:40681 every 8 connection(s)
I20260929 03:19:21.106976 1516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:21.110076 1584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:21.112195 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 1516
I20260929 03:19:21.112306 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:19:21.112584 14266 external_mini_cluster.cc:1428] Setting key 32ae0a0f8a586b62fd76fa1d6ba025a1
I20260929 03:19:21.116148 1584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebcb2fce6613442bb4891c277000fdf5: Bootstrap starting.
I20260929 03:19:21.118458 1584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebcb2fce6613442bb4891c277000fdf5: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:21.119210 1584 log.cc:824] T 00000000000000000000000000000000 P ebcb2fce6613442bb4891c277000fdf5: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:21.121171 1584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebcb2fce6613442bb4891c277000fdf5: No bootstrap required, opened a new log
I20260929 03:19:21.121804 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:21.114022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50914 (local address 127.13.238.190:40681)
0929 03:19:21.114709 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:21.114724 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:21.114760 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:19:21.115396 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:21.115423 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:21.115873 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:21.116209 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:21.116667 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.117677 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.118634 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.119131 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.119278 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:21.120071 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:21.120076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:21.120090 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:21.120475 (+ 385us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:21.120534 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:21.120554 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:21.120640 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:21.120782 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 03:19:21.121048 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":158,"threads_started":1}
I20260929 03:19:21.124480 1584 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebcb2fce6613442bb4891c277000fdf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcb2fce6613442bb4891c277000fdf5" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 40681 } }
I20260929 03:19:21.124706 1584 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebcb2fce6613442bb4891c277000fdf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:21.124761 1584 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebcb2fce6613442bb4891c277000fdf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebcb2fce6613442bb4891c277000fdf5, State: Initialized, Role: FOLLOWER
I20260929 03:19:21.125156 1584 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebcb2fce6613442bb4891c277000fdf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcb2fce6613442bb4891c277000fdf5" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 40681 } }
I20260929 03:19:21.125283 1584 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebcb2fce6613442bb4891c277000fdf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:21.125348 1584 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebcb2fce6613442bb4891c277000fdf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:21.125453 1584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebcb2fce6613442bb4891c277000fdf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:21.126372 1584 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebcb2fce6613442bb4891c277000fdf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcb2fce6613442bb4891c277000fdf5" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 40681 } }
I20260929 03:19:21.126691 1584 leader_election.cc:302] T 00000000000000000000000000000000 P ebcb2fce6613442bb4891c277000fdf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebcb2fce6613442bb4891c277000fdf5; no voters:
I20260929 03:19:21.126986 1584 leader_election.cc:288] T 00000000000000000000000000000000 P ebcb2fce6613442bb4891c277000fdf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:19:21.127067 1589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebcb2fce6613442bb4891c277000fdf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:21.127293 1589 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebcb2fce6613442bb4891c277000fdf5 [term 1 LEADER]: Becoming Leader. State: Replica: ebcb2fce6613442bb4891c277000fdf5, State: Running, Role: LEADER
I20260929 03:19:21.127622 1589 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebcb2fce6613442bb4891c277000fdf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcb2fce6613442bb4891c277000fdf5" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 40681 } }
I20260929 03:19:21.128094 1584 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebcb2fce6613442bb4891c277000fdf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:21.129315 1591 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebcb2fce6613442bb4891c277000fdf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebcb2fce6613442bb4891c277000fdf5. Latest consensus state: current_term: 1 leader_uuid: "ebcb2fce6613442bb4891c277000fdf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcb2fce6613442bb4891c277000fdf5" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 40681 } } }
I20260929 03:19:21.129343 1590 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebcb2fce6613442bb4891c277000fdf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebcb2fce6613442bb4891c277000fdf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcb2fce6613442bb4891c277000fdf5" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 40681 } } }
I20260929 03:19:21.129431 1591 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebcb2fce6613442bb4891c277000fdf5 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:21.129436 1590 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebcb2fce6613442bb4891c277000fdf5 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:21.129849 1599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:21.133091 1599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:21.138485 1599 catalog_manager.cc:1347] Generated new cluster ID: 156df4d4c29740a79016c184e875601e
I20260929 03:19:21.138541 1599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:21.152019 1599 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:21.152935 1599 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:21.159652 1599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebcb2fce6613442bb4891c277000fdf5: Generated new TSK 0
I20260929 03:19:21.160338 1599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:19:21.167925 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.129:0
--local_ip_for_outbound_sockets=127.13.238.129
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:40681
--builtin_ntp_servers=127.13.238.148:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:19:21.281651 1608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:21.281925 1608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:21.281971 1608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:21.286058 1608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:21.286131 1608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:21.286187 1608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.129
I20260929 03:19:21.291054 1608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.238.129
--webserver_port=0
--tserver_master_addrs=127.13.238.190:40681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.129
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:21.292153 1608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:21.293591 1608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:21.302891 1616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:21.302896 1614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:21.302891 1613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:21.303278 1608 server_base.cc:1034] running on GCE node
I20260929 03:19:21.303689 1608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:21.304342 1608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:21.305511 1608 hybrid_clock.cc:648] HybridClock initialized: now 1790651961305481 us; error 48 us; skew 500 ppm
I20260929 03:19:21.307317 1608 webserver.cc:460] Webserver started at http://127.13.238.129:40013/ using document root <none> and password file <none>
I20260929 03:19:21.307893 1608 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:21.307956 1608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:21.308143 1608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:21.310288 1608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-0/data/instance:
uuid: "c4ceae78eca04af78e8b4e68545e86fe"
format_stamp: "Formatted at 2026-09-29 03:19:21 on dist-test-slave-nbjg"
server_key: "f8bc501b9cb424f554b095b8a68e8813"
server_key_iv: "d0e28b1af2dcd89ec7509093de2d3bca"
server_key_version: "encryptionkey@0"
I20260929 03:19:21.310843 1608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance:
uuid: "c4ceae78eca04af78e8b4e68545e86fe"
format_stamp: "Formatted at 2026-09-29 03:19:21 on dist-test-slave-nbjg"
server_key: "f8bc501b9cb424f554b095b8a68e8813"
server_key_iv: "d0e28b1af2dcd89ec7509093de2d3bca"
server_key_version: "encryptionkey@0"
I20260929 03:19:21.314450 1608 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:19:21.316844 1622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:21.318051 1608 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:19:21.318157 1608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-0/wal
uuid: "c4ceae78eca04af78e8b4e68545e86fe"
format_stamp: "Formatted at 2026-09-29 03:19:21 on dist-test-slave-nbjg"
server_key: "f8bc501b9cb424f554b095b8a68e8813"
server_key_iv: "d0e28b1af2dcd89ec7509093de2d3bca"
server_key_version: "encryptionkey@0"
I20260929 03:19:21.318281 1608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:21.344024 1608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:21.344732 1608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:21.344901 1608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:21.345497 1608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:21.346486 1608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:21.346534 1608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:21.346591 1608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:21.346616 1608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:21.355013 1608 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.129:42457
I20260929 03:19:21.355098 1735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.129:42457 every 8 connection(s)
I20260929 03:19:21.355875 1608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-0/data/info.pb
I20260929 03:19:21.364287 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 1608
I20260929 03:19:21.364413 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-0/wal/instance
I20260929 03:19:21.364511 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:21.357643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.129:49477 (local address 127.13.238.190:40681)
0929 03:19:21.357773 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:21.357776 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:21.358835 (+ 1059us) server_negotiation.cc:407] Connection header received
0929 03:19:21.360194 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:21.360197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:21.360245 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:21.360315 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:21.361133 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.361668 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.362781 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.363006 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.363083 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:21.364039 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:21.364042 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:21.364044 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:21.364055 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:21.364075 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:21.364082 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:21.364136 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:21.364295 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 03:19:21.364345 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:19:21.364713 14266 external_mini_cluster.cc:1428] Setting key d2967a31b69e0edf7e9abf928ca4a239
I20260929 03:19:21.365527 1738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:21.358030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:40681 (local address 127.13.238.129:49477)
0929 03:19:21.358691 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:21.358733 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:21.359982 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:21.360450 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:21.360463 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:21.360549 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:21.361025 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:21.361037 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.361758 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.361760 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:21.362676 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:21.362686 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.363124 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.363126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:21.363259 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:21.363783 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:19:21.363785 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:21.363824 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:19:21.363886 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:21.364156 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:21.364164 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:21.364170 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:19:21.364299 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 03:19:21.364609 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":121,"threads_started":1}
I20260929 03:19:21.366684 1736 heartbeater.cc:347] Connected to a master server at 127.13.238.190:40681
I20260929 03:19:21.366765 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.130:0
--local_ip_for_outbound_sockets=127.13.238.130
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:40681
--builtin_ntp_servers=127.13.238.148:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:19:21.366925 1736 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:21.367441 1736 heartbeater.cc:511] Master 127.13.238.190:40681 requested a full tablet report, sending...
I20260929 03:19:21.368875 1547 ts_manager.cc:194] Registered new tserver with Master: c4ceae78eca04af78e8b4e68545e86fe (127.13.238.129:42457)
I20260929 03:19:21.370018 1547 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.238.129:49477
W20260929 03:19:21.476926 1739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:21.477221 1739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:21.477262 1739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:21.481549 1739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:21.481609 1739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:21.481666 1739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.130
I20260929 03:19:21.486418 1739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.238.130
--webserver_port=0
--tserver_master_addrs=127.13.238.190:40681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.130
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:21.487568 1739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:21.488952 1739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:19:21.498415 1739 server_base.cc:1034] running on GCE node
W20260929 03:19:21.498344 1745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:21.498445 1744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:21.498344 1747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:21.498996 1739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:21.499661 1739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:21.500828 1739 hybrid_clock.cc:648] HybridClock initialized: now 1790651961500799 us; error 42 us; skew 500 ppm
I20260929 03:19:21.502767 1739 webserver.cc:460] Webserver started at http://127.13.238.130:38045/ using document root <none> and password file <none>
I20260929 03:19:21.503378 1739 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:21.503446 1739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:21.503671 1739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:21.505818 1739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-1/data/instance:
uuid: "0325358b3a0645ec8a2b9b3924e85a20"
format_stamp: "Formatted at 2026-09-29 03:19:21 on dist-test-slave-nbjg"
server_key: "a588a61981891c285d80ede86cabfeed"
server_key_iv: "3740d7d4d19638a5420acc3b933e87c5"
server_key_version: "encryptionkey@0"
I20260929 03:19:21.506399 1739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance:
uuid: "0325358b3a0645ec8a2b9b3924e85a20"
format_stamp: "Formatted at 2026-09-29 03:19:21 on dist-test-slave-nbjg"
server_key: "a588a61981891c285d80ede86cabfeed"
server_key_iv: "3740d7d4d19638a5420acc3b933e87c5"
server_key_version: "encryptionkey@0"
I20260929 03:19:21.510129 1739 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:19:21.512665 1753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:21.514037 1739 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:19:21.514158 1739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-1/wal
uuid: "0325358b3a0645ec8a2b9b3924e85a20"
format_stamp: "Formatted at 2026-09-29 03:19:21 on dist-test-slave-nbjg"
server_key: "a588a61981891c285d80ede86cabfeed"
server_key_iv: "3740d7d4d19638a5420acc3b933e87c5"
server_key_version: "encryptionkey@0"
I20260929 03:19:21.514300 1739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:21.540863 1739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:21.541606 1739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:21.541785 1739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:21.542451 1739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:21.543445 1739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:21.543493 1739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:21.543548 1739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:21.543577 1739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:21.552345 1739 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.130:38243
I20260929 03:19:21.552435 1866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.130:38243 every 8 connection(s)
I20260929 03:19:21.553318 1739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-1/data/info.pb
I20260929 03:19:21.554548 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 1739
I20260929 03:19:21.554653 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-1/wal/instance
I20260929 03:19:21.554924 14266 external_mini_cluster.cc:1428] Setting key 8fa28c33aba3360277aac7c24681d4c7
I20260929 03:19:21.557156 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.238.131:0
--local_ip_for_outbound_sockets=127.13.238.131
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:40681
--builtin_ntp_servers=127.13.238.148:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:19:21.563995 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:21.555299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:51955 (local address 127.13.238.190:40681)
0929 03:19:21.555441 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:21.555445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:21.556714 (+ 1269us) server_negotiation.cc:407] Connection header received
0929 03:19:21.558186 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:21.558212 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:21.558274 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:21.558362 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:21.559552 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.560090 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.561772 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.562003 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.562087 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:21.563443 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:21.563446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:21.563448 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:21.563461 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:21.563481 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:21.563489 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:21.563547 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:21.563769 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 03:19:21.563830 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:19:21.565136 1869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:21.555792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:40681 (local address 127.13.238.130:51955)
0929 03:19:21.556588 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:21.556633 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:21.557962 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:21.558576 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:21.558585 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:21.558702 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:21.559385 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:21.559404 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.560205 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.560209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:21.561634 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:21.561646 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.562165 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.562170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:21.562423 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:21.563109 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:19:21.563111 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:21.563172 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:19:21.563263 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:21.563570 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:21.563579 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:21.563586 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:19:21.563725 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 03:19:21.564047 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":113,"threads_started":1}
I20260929 03:19:21.566520 1867 heartbeater.cc:347] Connected to a master server at 127.13.238.190:40681
I20260929 03:19:21.566777 1867 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:21.567317 1867 heartbeater.cc:511] Master 127.13.238.190:40681 requested a full tablet report, sending...
I20260929 03:19:21.568401 1547 ts_manager.cc:194] Registered new tserver with Master: 0325358b3a0645ec8a2b9b3924e85a20 (127.13.238.130:38243)
I20260929 03:19:21.568964 1547 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.238.130:51955
W20260929 03:19:21.672114 1870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:21.672401 1870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:21.672456 1870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:21.676576 1870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:21.676638 1870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:21.676712 1870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.238.131
I20260929 03:19:21.681679 1870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.238.131
--webserver_port=0
--tserver_master_addrs=127.13.238.190:40681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.238.131
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:21.682866 1870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:21.684319 1870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:21.693851 1876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:21.693858 1875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:21.693859 1878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:21.693912 1870 server_base.cc:1034] running on GCE node
I20260929 03:19:21.694545 1870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:21.695310 1870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:21.696485 1870 hybrid_clock.cc:648] HybridClock initialized: now 1790651961696453 us; error 45 us; skew 500 ppm
I20260929 03:19:21.702020 1870 webserver.cc:460] Webserver started at http://127.13.238.131:36263/ using document root <none> and password file <none>
I20260929 03:19:21.702658 1870 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:21.702723 1870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:21.702895 1870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:21.705003 1870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-2/data/instance:
uuid: "7d779eb867f849539a017fe1c2dccaad"
format_stamp: "Formatted at 2026-09-29 03:19:21 on dist-test-slave-nbjg"
server_key: "2926129f7435ef7dfd1b8b4ca62059e3"
server_key_iv: "5d7f87ed59bebc5bc7a6959ca2cecad9"
server_key_version: "encryptionkey@0"
I20260929 03:19:21.705560 1870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance:
uuid: "7d779eb867f849539a017fe1c2dccaad"
format_stamp: "Formatted at 2026-09-29 03:19:21 on dist-test-slave-nbjg"
server_key: "2926129f7435ef7dfd1b8b4ca62059e3"
server_key_iv: "5d7f87ed59bebc5bc7a6959ca2cecad9"
server_key_version: "encryptionkey@0"
I20260929 03:19:21.709331 1870 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:19:21.711826 1884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:21.713037 1870 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 03:19:21.713151 1870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-2/wal
uuid: "7d779eb867f849539a017fe1c2dccaad"
format_stamp: "Formatted at 2026-09-29 03:19:21 on dist-test-slave-nbjg"
server_key: "2926129f7435ef7dfd1b8b4ca62059e3"
server_key_iv: "5d7f87ed59bebc5bc7a6959ca2cecad9"
server_key_version: "encryptionkey@0"
I20260929 03:19:21.713263 1870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:21.736325 1870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:21.737080 1870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:21.737241 1870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:21.737850 1870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:19:21.738900 1870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:19:21.738946 1870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:21.738997 1870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:19:21.739019 1870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:21.747613 1870 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.131:37621
I20260929 03:19:21.747650 1997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.131:37621 every 8 connection(s)
I20260929 03:19:21.748551 1870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-2/data/info.pb
I20260929 03:19:21.753901 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 1870
I20260929 03:19:21.754001 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-2/wal/instance
I20260929 03:19:21.754282 14266 external_mini_cluster.cc:1428] Setting key 030c38b55e1fc557d731a1668c0a73c9
I20260929 03:19:21.758693 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:21.750486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.131:40639 (local address 127.13.238.190:40681)
0929 03:19:21.750676 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:21.750681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:21.751743 (+ 1062us) server_negotiation.cc:407] Connection header received
0929 03:19:21.753350 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:21.753354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:21.753422 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:21.753497 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:21.754555 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.755200 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.756759 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.757016 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.757107 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:21.758159 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:21.758163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:21.758165 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:21.758177 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:21.758225 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:21.758236 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:21.758307 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:21.758515 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 03:19:21.758578 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 03:19:21.759534 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:21.750917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:40681 (local address 127.13.238.131:40639)
0929 03:19:21.751592 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:21.751635 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:21.753112 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:21.753684 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:21.753691 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:21.753787 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:21.754397 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:21.754416 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.755334 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.755338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:21.756634 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:21.756644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.757153 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.757156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:21.757309 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:21.757892 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:19:21.757894 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:21.757937 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:19:21.758009 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:21.758306 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:21.758311 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:21.758315 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:19:21.758414 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 03:19:21.758677 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":107,"threads_started":1}
I20260929 03:19:21.760582 1998 heartbeater.cc:347] Connected to a master server at 127.13.238.190:40681
I20260929 03:19:21.760807 1998 heartbeater.cc:464] Registering TS with master...
I20260929 03:19:21.761355 1998 heartbeater.cc:511] Master 127.13.238.190:40681 requested a full tablet report, sending...
I20260929 03:19:21.762389 1547 ts_manager.cc:194] Registered new tserver with Master: 7d779eb867f849539a017fe1c2dccaad (127.13.238.131:37621)
I20260929 03:19:21.762935 1547 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.238.131:40639
I20260929 03:19:21.768491 14266 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:19:21.774070 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:21.770470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50928 (local address 127.13.238.190:40681)
0929 03:19:21.770617 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:21.770623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:21.770763 (+ 140us) server_negotiation.cc:407] Connection header received
0929 03:19:21.770884 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:21.770887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:21.770932 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:21.770997 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:21.771277 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.771804 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.772615 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.772838 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.772924 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:21.773671 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:21.773675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:21.773677 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:21.773690 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:21.773708 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:21.773717 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:21.773771 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:21.773870 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 03:19:21.773929 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 03:19:21.776834 1547 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:19:21.779922 1547 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:19:21.793553 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:21.788210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53580 (local address 127.13.238.130:38243)
0929 03:19:21.789380 (+ 1170us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:21.789396 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:21.789435 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:19:21.789518 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:21.789524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:21.789733 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:21.790124 (+ 391us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:21.790286 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.791042 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.792356 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.792891 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.793007 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:21.793195 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:21.793298 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 03:19:21.793396 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":100,"threads_started":1}
I20260929 03:19:21.793558 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:21.788132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.130:38243 (local address 127.0.0.1:53580)
0929 03:19:21.788978 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:21.789035 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:21.789162 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:21.789891 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:21.789897 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:21.789920 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:21.790079 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:21.790087 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.791156 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.791159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:21.792238 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:21.792248 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.793006 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.793009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:21.793121 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:21.793158 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 03:19:21.793270 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 03:19:21.793392 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":86,"threads_started":1}
I20260929 03:19:21.794569 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:21.788643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:37621 (local address 127.0.0.1:55122)
0929 03:19:21.789382 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:21.789393 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:21.789465 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:21.790068 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:21.790072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:21.790089 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:21.790239 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:21.790248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.791694 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.791697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:21.793036 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:21.793044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.794259 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.794262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:21.794329 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:21.794347 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 03:19:21.794395 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 03:19:21.794454 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":261,"thread_start_us":89,"threads_started":1}
I20260929 03:19:21.794749 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:21.789031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55122 (local address 127.13.238.131:37621)
0929 03:19:21.789425 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:21.789435 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:21.789455 (+ 20us) server_negotiation.cc:407] Connection header received
0929 03:19:21.789580 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:21.789584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:21.789732 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:21.789889 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:21.790361 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.791563 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.793150 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.794131 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.794274 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:21.794424 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:21.794471 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 03:19:21.794571 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":149,"threads_started":1}
I20260929 03:19:21.795059 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:21.788169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:42457 (local address 127.0.0.1:48052)
0929 03:19:21.789072 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:21.789085 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:21.789164 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:21.790115 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:21.790118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:21.790135 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:21.790267 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:21.790273 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.791723 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.791726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:21.792873 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:21.792884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.794722 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.794726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:21.794813 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:21.794836 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:19:21.794887 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 03:19:21.794950 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"mutex_wait_us":4,"thread_start_us":119,"threads_started":1}
I20260929 03:19:21.795759 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:21.788541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48052 (local address 127.13.238.129:42457)
0929 03:19:21.789700 (+ 1159us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:21.789708 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:21.789727 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:19:21.789786 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:21.789790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:21.789973 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:21.790402 (+ 429us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:21.790545 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.791621 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.793025 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.794597 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.795303 (+ 706us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:21.795466 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:21.795509 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 03:19:21.795611 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1069,"thread_start_us":100,"threads_started":1}
I20260929 03:19:21.796826 1801 tablet_service.cc:1467] Processing CreateTablet for tablet 58f65e02d6b5440488c46b91803a3fe4 (DEFAULT_TABLE table=test-table [id=3acdbed307934acdb6135211b422c0fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:19:21.798038 1801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58f65e02d6b5440488c46b91803a3fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:21.797745 1932 tablet_service.cc:1467] Processing CreateTablet for tablet 58f65e02d6b5440488c46b91803a3fe4 (DEFAULT_TABLE table=test-table [id=3acdbed307934acdb6135211b422c0fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:19:21.798629 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58f65e02d6b5440488c46b91803a3fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:21.798375 1670 tablet_service.cc:1467] Processing CreateTablet for tablet 58f65e02d6b5440488c46b91803a3fe4 (DEFAULT_TABLE table=test-table [id=3acdbed307934acdb6135211b422c0fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:19:21.799350 1670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58f65e02d6b5440488c46b91803a3fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:21.804453 2015 tablet_bootstrap.cc:492] T 58f65e02d6b5440488c46b91803a3fe4 P 7d779eb867f849539a017fe1c2dccaad: Bootstrap starting.
I20260929 03:19:21.805897 2017 tablet_bootstrap.cc:492] T 58f65e02d6b5440488c46b91803a3fe4 P c4ceae78eca04af78e8b4e68545e86fe: Bootstrap starting.
I20260929 03:19:21.806308 2016 tablet_bootstrap.cc:492] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20: Bootstrap starting.
I20260929 03:19:21.806972 2015 tablet_bootstrap.cc:654] T 58f65e02d6b5440488c46b91803a3fe4 P 7d779eb867f849539a017fe1c2dccaad: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:21.807833 2015 log.cc:824] T 58f65e02d6b5440488c46b91803a3fe4 P 7d779eb867f849539a017fe1c2dccaad: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:21.808055 2016 tablet_bootstrap.cc:654] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:21.808228 2017 tablet_bootstrap.cc:654] T 58f65e02d6b5440488c46b91803a3fe4 P c4ceae78eca04af78e8b4e68545e86fe: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:21.808807 2016 log.cc:824] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:21.808950 2017 log.cc:824] T 58f65e02d6b5440488c46b91803a3fe4 P c4ceae78eca04af78e8b4e68545e86fe: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:21.810128 2015 tablet_bootstrap.cc:492] T 58f65e02d6b5440488c46b91803a3fe4 P 7d779eb867f849539a017fe1c2dccaad: No bootstrap required, opened a new log
I20260929 03:19:21.810360 2015 ts_tablet_manager.cc:1397] T 58f65e02d6b5440488c46b91803a3fe4 P 7d779eb867f849539a017fe1c2dccaad: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 03:19:21.811235 2016 tablet_bootstrap.cc:492] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20: No bootstrap required, opened a new log
I20260929 03:19:21.811360 2017 tablet_bootstrap.cc:492] T 58f65e02d6b5440488c46b91803a3fe4 P c4ceae78eca04af78e8b4e68545e86fe: No bootstrap required, opened a new log
I20260929 03:19:21.811415 2016 ts_tablet_manager.cc:1397] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 03:19:21.811540 2017 ts_tablet_manager.cc:1397] T 58f65e02d6b5440488c46b91803a3fe4 P c4ceae78eca04af78e8b4e68545e86fe: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 03:19:21.814337 2016 raft_consensus.cc:348] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0325358b3a0645ec8a2b9b3924e85a20" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 38243 } } peers { permanent_uuid: "c4ceae78eca04af78e8b4e68545e86fe" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42457 } } peers { permanent_uuid: "7d779eb867f849539a017fe1c2dccaad" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 37621 } }
I20260929 03:19:21.814589 2016 raft_consensus.cc:374] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:21.814642 2016 raft_consensus.cc:729] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0325358b3a0645ec8a2b9b3924e85a20, State: Initialized, Role: FOLLOWER
I20260929 03:19:21.814719 2015 raft_consensus.cc:348] T 58f65e02d6b5440488c46b91803a3fe4 P 7d779eb867f849539a017fe1c2dccaad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0325358b3a0645ec8a2b9b3924e85a20" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 38243 } } peers { permanent_uuid: "c4ceae78eca04af78e8b4e68545e86fe" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42457 } } peers { permanent_uuid: "7d779eb867f849539a017fe1c2dccaad" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 37621 } }
I20260929 03:19:21.815001 2015 raft_consensus.cc:374] T 58f65e02d6b5440488c46b91803a3fe4 P 7d779eb867f849539a017fe1c2dccaad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:21.815068 2015 raft_consensus.cc:729] T 58f65e02d6b5440488c46b91803a3fe4 P 7d779eb867f849539a017fe1c2dccaad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d779eb867f849539a017fe1c2dccaad, State: Initialized, Role: FOLLOWER
I20260929 03:19:21.815119 2016 consensus_queue.cc:260] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0325358b3a0645ec8a2b9b3924e85a20" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 38243 } } peers { permanent_uuid: "c4ceae78eca04af78e8b4e68545e86fe" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42457 } } peers { permanent_uuid: "7d779eb867f849539a017fe1c2dccaad" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 37621 } }
I20260929 03:19:21.815368 2017 raft_consensus.cc:348] T 58f65e02d6b5440488c46b91803a3fe4 P c4ceae78eca04af78e8b4e68545e86fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0325358b3a0645ec8a2b9b3924e85a20" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 38243 } } peers { permanent_uuid: "c4ceae78eca04af78e8b4e68545e86fe" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42457 } } peers { permanent_uuid: "7d779eb867f849539a017fe1c2dccaad" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 37621 } }
I20260929 03:19:21.815619 2017 raft_consensus.cc:374] T 58f65e02d6b5440488c46b91803a3fe4 P c4ceae78eca04af78e8b4e68545e86fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:21.815577 2015 consensus_queue.cc:260] T 58f65e02d6b5440488c46b91803a3fe4 P 7d779eb867f849539a017fe1c2dccaad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0325358b3a0645ec8a2b9b3924e85a20" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 38243 } } peers { permanent_uuid: "c4ceae78eca04af78e8b4e68545e86fe" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42457 } } peers { permanent_uuid: "7d779eb867f849539a017fe1c2dccaad" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 37621 } }
I20260929 03:19:21.815672 2017 raft_consensus.cc:729] T 58f65e02d6b5440488c46b91803a3fe4 P c4ceae78eca04af78e8b4e68545e86fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4ceae78eca04af78e8b4e68545e86fe, State: Initialized, Role: FOLLOWER
I20260929 03:19:21.816144 2017 consensus_queue.cc:260] T 58f65e02d6b5440488c46b91803a3fe4 P c4ceae78eca04af78e8b4e68545e86fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0325358b3a0645ec8a2b9b3924e85a20" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 38243 } } peers { permanent_uuid: "c4ceae78eca04af78e8b4e68545e86fe" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42457 } } peers { permanent_uuid: "7d779eb867f849539a017fe1c2dccaad" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 37621 } }
I20260929 03:19:21.816233 2016 ts_tablet_manager.cc:1428] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 03:19:21.816650 2015 ts_tablet_manager.cc:1428] T 58f65e02d6b5440488c46b91803a3fe4 P 7d779eb867f849539a017fe1c2dccaad: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 03:19:21.817090 1998 heartbeater.cc:503] Master 127.13.238.190:40681 was elected leader, sending a full tablet report...
I20260929 03:19:21.817121 1867 heartbeater.cc:503] Master 127.13.238.190:40681 was elected leader, sending a full tablet report...
I20260929 03:19:21.817405 1736 heartbeater.cc:503] Master 127.13.238.190:40681 was elected leader, sending a full tablet report...
I20260929 03:19:21.818691 2017 ts_tablet_manager.cc:1428] T 58f65e02d6b5440488c46b91803a3fe4 P c4ceae78eca04af78e8b4e68545e86fe: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260929 03:19:21.818734 2021 raft_consensus.cc:482] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:19:21.818988 2021 raft_consensus.cc:504] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0325358b3a0645ec8a2b9b3924e85a20" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 38243 } } peers { permanent_uuid: "c4ceae78eca04af78e8b4e68545e86fe" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42457 } } peers { permanent_uuid: "7d779eb867f849539a017fe1c2dccaad" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 37621 } }
I20260929 03:19:21.823010 2021 leader_election.cc:288] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4ceae78eca04af78e8b4e68545e86fe (127.13.238.129:42457), 7d779eb867f849539a017fe1c2dccaad (127.13.238.131:37621)
I20260929 03:19:21.824642 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:21.820727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:50553 (local address 127.13.238.129:42457)
0929 03:19:21.820834 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:21.820839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:21.821209 (+ 370us) server_negotiation.cc:407] Connection header received
0929 03:19:21.821333 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:21.821336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:21.821396 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:21.821469 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:21.821737 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.822481 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.823760 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.824136 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.824236 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:21.824287 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:21.824457 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 03:19:21.824546 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:19:21.824645 1869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:21.820651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:42457 (local address 127.13.238.130:50553)
0929 03:19:21.821149 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:21.821163 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:21.821243 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:21.821487 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:21.821490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:21.821505 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:21.821616 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:21.821624 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.822618 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.822621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:21.823646 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:21.823656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.824246 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.824249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:21.824344 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:21.824367 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:19:21.824432 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 03:19:21.824503 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":3,"thread_start_us":91,"threads_started":1}
I20260929 03:19:21.824864 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:21.821385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.238.130:37243 (local address 127.13.238.131:37621)
0929 03:19:21.821497 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:21.821501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:21.821512 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:19:21.821553 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:21.821555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:21.821599 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:21.821664 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:19:21.822053 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.822758 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.823947 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.824326 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.824423 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:21.824467 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:21.824666 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 03:19:21.824743 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:19:21.825191 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:21.820502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.131:37621 (local address 127.13.238.130:37243)
0929 03:19:21.820980 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:21.820995 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:21.821090 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:21.821765 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:21.821768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:21.821784 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:19:21.821910 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:21.821916 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.822873 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.822876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:21.823824 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:21.823834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.824441 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.824444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:21.824533 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:21.824556 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:19:21.824976 (+ 420us) client_negotiation.cc:240] Negotiation successful
0929 03:19:21.825064 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":20}
I20260929 03:19:21.825230 1690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58f65e02d6b5440488c46b91803a3fe4" candidate_uuid: "0325358b3a0645ec8a2b9b3924e85a20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4ceae78eca04af78e8b4e68545e86fe" is_pre_election: true
I20260929 03:19:21.825554 1690 raft_consensus.cc:2454] T 58f65e02d6b5440488c46b91803a3fe4 P c4ceae78eca04af78e8b4e68545e86fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0325358b3a0645ec8a2b9b3924e85a20 in term 0.
I20260929 03:19:21.825860 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58f65e02d6b5440488c46b91803a3fe4" candidate_uuid: "0325358b3a0645ec8a2b9b3924e85a20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d779eb867f849539a017fe1c2dccaad" is_pre_election: true
I20260929 03:19:21.826012 1754 leader_election.cc:302] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0325358b3a0645ec8a2b9b3924e85a20, c4ceae78eca04af78e8b4e68545e86fe; no voters:
I20260929 03:19:21.826144 1952 raft_consensus.cc:2454] T 58f65e02d6b5440488c46b91803a3fe4 P 7d779eb867f849539a017fe1c2dccaad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0325358b3a0645ec8a2b9b3924e85a20 in term 0.
I20260929 03:19:21.826298 2021 raft_consensus.cc:2781] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:19:21.826372 2021 raft_consensus.cc:482] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:19:21.826421 2021 raft_consensus.cc:3037] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:21.827382 2021 raft_consensus.cc:504] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0325358b3a0645ec8a2b9b3924e85a20" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 38243 } } peers { permanent_uuid: "c4ceae78eca04af78e8b4e68545e86fe" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42457 } } peers { permanent_uuid: "7d779eb867f849539a017fe1c2dccaad" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 37621 } }
I20260929 03:19:21.827718 2021 leader_election.cc:288] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20 [CANDIDATE]: Term 1 election: Requested vote from peers c4ceae78eca04af78e8b4e68545e86fe (127.13.238.129:42457), 7d779eb867f849539a017fe1c2dccaad (127.13.238.131:37621)
I20260929 03:19:21.827993 1690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58f65e02d6b5440488c46b91803a3fe4" candidate_uuid: "0325358b3a0645ec8a2b9b3924e85a20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4ceae78eca04af78e8b4e68545e86fe"
I20260929 03:19:21.828035 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58f65e02d6b5440488c46b91803a3fe4" candidate_uuid: "0325358b3a0645ec8a2b9b3924e85a20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d779eb867f849539a017fe1c2dccaad"
I20260929 03:19:21.828097 1690 raft_consensus.cc:3037] T 58f65e02d6b5440488c46b91803a3fe4 P c4ceae78eca04af78e8b4e68545e86fe [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:21.828136 1952 raft_consensus.cc:3037] T 58f65e02d6b5440488c46b91803a3fe4 P 7d779eb867f849539a017fe1c2dccaad [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:21.829200 1690 raft_consensus.cc:2454] T 58f65e02d6b5440488c46b91803a3fe4 P c4ceae78eca04af78e8b4e68545e86fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0325358b3a0645ec8a2b9b3924e85a20 in term 1.
I20260929 03:19:21.829303 1952 raft_consensus.cc:2454] T 58f65e02d6b5440488c46b91803a3fe4 P 7d779eb867f849539a017fe1c2dccaad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0325358b3a0645ec8a2b9b3924e85a20 in term 1.
I20260929 03:19:21.829521 1754 leader_election.cc:302] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0325358b3a0645ec8a2b9b3924e85a20, c4ceae78eca04af78e8b4e68545e86fe; no voters:
I20260929 03:19:21.829685 2021 raft_consensus.cc:2781] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:21.829813 2021 raft_consensus.cc:686] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20 [term 1 LEADER]: Becoming Leader. State: Replica: 0325358b3a0645ec8a2b9b3924e85a20, State: Running, Role: LEADER
I20260929 03:19:21.830128 2021 consensus_queue.cc:237] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0325358b3a0645ec8a2b9b3924e85a20" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 38243 } } peers { permanent_uuid: "c4ceae78eca04af78e8b4e68545e86fe" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42457 } } peers { permanent_uuid: "7d779eb867f849539a017fe1c2dccaad" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 37621 } }
I20260929 03:19:21.833066 1547 catalog_manager.cc:5495] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0325358b3a0645ec8a2b9b3924e85a20 (127.13.238.130). New cstate: current_term: 1 leader_uuid: "0325358b3a0645ec8a2b9b3924e85a20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0325358b3a0645ec8a2b9b3924e85a20" member_type: VOTER last_known_addr { host: "127.13.238.130" port: 38243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4ceae78eca04af78e8b4e68545e86fe" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d779eb867f849539a017fe1c2dccaad" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 37621 } health_report { overall_health: UNKNOWN } } }
I20260929 03:19:21.843492 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:21.840135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53588 (local address 127.13.238.130:38243)
0929 03:19:21.840251 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:21.840256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:21.840269 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:19:21.840405 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:21.840407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:21.840459 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:21.840558 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:19:21.840788 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.841274 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.842106 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:21.842347 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:21.842429 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:21.842721 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:19:21.843130 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:19:21.843236 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:21.843305 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 03:19:21.843359 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 03:19:21.849126 1690 raft_consensus.cc:1261] T 58f65e02d6b5440488c46b91803a3fe4 P c4ceae78eca04af78e8b4e68545e86fe [term 1 FOLLOWER]: Refusing update from remote peer 0325358b3a0645ec8a2b9b3924e85a20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:19:21.849144 1952 raft_consensus.cc:1261] T 58f65e02d6b5440488c46b91803a3fe4 P 7d779eb867f849539a017fe1c2dccaad [term 1 FOLLOWER]: Refusing update from remote peer 0325358b3a0645ec8a2b9b3924e85a20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:19:21.849781 2024 consensus_queue.cc:1035] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4ceae78eca04af78e8b4e68545e86fe" member_type: VOTER last_known_addr { host: "127.13.238.129" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:19:21.849956 2021 consensus_queue.cc:1035] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d779eb867f849539a017fe1c2dccaad" member_type: VOTER last_known_addr { host: "127.13.238.131" port: 37621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 03:19:21.856959 1737 tablet.cc:2369] T 58f65e02d6b5440488c46b91803a3fe4 P c4ceae78eca04af78e8b4e68545e86fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:19:21.857180 2028 mvcc.cc:205] Tried to move back new op lower bound from 7334510435727237120 to 7334510435658596352. Current Snapshot: MvccSnapshot[applied={T|T < 7334510435727237120}]
I20260929 03:19:21.859481 2030 mvcc.cc:205] Tried to move back new op lower bound from 7334510435727237120 to 7334510435658596352. Current Snapshot: MvccSnapshot[applied={T|T < 7334510435727237120}]
I20260929 03:19:21.859833 2029 mvcc.cc:205] Tried to move back new op lower bound from 7334510435727237120 to 7334510435658596352. Current Snapshot: MvccSnapshot[applied={T|T < 7334510435727237120 or (T in {7334510435727237120})}]
I20260929 03:19:21.866639 1546 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:19:21.866820 1546 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:19:21.869306 1546 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebcb2fce6613442bb4891c277000fdf5: Sending DeleteTablet for 3 replicas of tablet 58f65e02d6b5440488c46b91803a3fe4
I20260929 03:19:21.870175 1801 tablet_service.cc:1514] Processing DeleteTablet for tablet 58f65e02d6b5440488c46b91803a3fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:19:21 UTC) from {username='slave'} at 127.0.0.1:53580
I20260929 03:19:21.870280 1932 tablet_service.cc:1514] Processing DeleteTablet for tablet 58f65e02d6b5440488c46b91803a3fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:19:21 UTC) from {username='slave'} at 127.0.0.1:55122
I20260929 03:19:21.870708 2038 tablet_replica.cc:330] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20: stopping tablet replica
I20260929 03:19:21.870718 1670 tablet_service.cc:1514] Processing DeleteTablet for tablet 58f65e02d6b5440488c46b91803a3fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:19:21 UTC) from {username='slave'} at 127.0.0.1:48052
I20260929 03:19:21.871140 2038 raft_consensus.cc:2229] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:19:21.871260 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 1608
I20260929 03:19:21.872607 2038 raft_consensus.cc:2258] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:19:21.874065 2038 ts_tablet_manager.cc:1905] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:19:21.876998 2038 ts_tablet_manager.cc:1918] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:19:21.877094 2038 log.cc:1196] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-1/wal/wals/58f65e02d6b5440488c46b91803a3fe4
I20260929 03:19:21.877386 2038 ts_tablet_manager.cc:1939] T 58f65e02d6b5440488c46b91803a3fe4 P 0325358b3a0645ec8a2b9b3924e85a20: Deleting consensus metadata
W20260929 03:19:21.878144 1531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.238.129:42457 (error 108)
I20260929 03:19:21.878543 2039 tablet_replica.cc:330] T 58f65e02d6b5440488c46b91803a3fe4 P 7d779eb867f849539a017fe1c2dccaad: stopping tablet replica
I20260929 03:19:21.878729 1534 catalog_manager.cc:4841] TS 0325358b3a0645ec8a2b9b3924e85a20 (127.13.238.130:38243): tablet 58f65e02d6b5440488c46b91803a3fe4 (table test-table [id=3acdbed307934acdb6135211b422c0fc]) successfully deleted
I20260929 03:19:21.878895 2039 raft_consensus.cc:2229] T 58f65e02d6b5440488c46b91803a3fe4 P 7d779eb867f849539a017fe1c2dccaad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:19:21.879037 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 1739
I20260929 03:19:21.879271 2039 raft_consensus.cc:2258] T 58f65e02d6b5440488c46b91803a3fe4 P 7d779eb867f849539a017fe1c2dccaad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:19:21.880838 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:21.880534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.129:42457 (local address 127.0.0.1:48054)
0929 03:19:21.880675 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:21.880757 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.129:42457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 03:19:21.881078 1531 catalog_manager.cc:4584] TS c4ceae78eca04af78e8b4e68545e86fe (127.13.238.129:42457): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 58f65e02d6b5440488c46b91803a3fe4: Network error: Client connection negotiation failed: client connection to 127.13.238.129:42457: connect: Connection refused (error 111)
I20260929 03:19:21.882295 2039 ts_tablet_manager.cc:1905] T 58f65e02d6b5440488c46b91803a3fe4 P 7d779eb867f849539a017fe1c2dccaad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:19:21.884567 2039 ts_tablet_manager.cc:1918] T 58f65e02d6b5440488c46b91803a3fe4 P 7d779eb867f849539a017fe1c2dccaad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:19:21.884653 2039 log.cc:1196] T 58f65e02d6b5440488c46b91803a3fe4 P 7d779eb867f849539a017fe1c2dccaad: Deleting WAL directory at /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790651713532531-14266-0/minicluster-data/ts-2/wal/wals/58f65e02d6b5440488c46b91803a3fe4
I20260929 03:19:21.884902 2039 ts_tablet_manager.cc:1939] T 58f65e02d6b5440488c46b91803a3fe4 P 7d779eb867f849539a017fe1c2dccaad: Deleting consensus metadata
I20260929 03:19:21.885656 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 1870
I20260929 03:19:21.890857 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 1516
2026-09-29T03:19:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (992 ms)
[----------] 6 tests from EncryptionPolicyTest (10296 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:19:21 dist-test-slave-nbjg krb5kdc[2044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:19:21 dist-test-slave-nbjg krb5kdc[2044](info): set up 2 sockets
Sep 29 03:19:21 dist-test-slave-nbjg krb5kdc[2044](info): commencing operation
krb5kdc: starting...
W20260929 03:19:22.850386 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:19:22 dist-test-slave-nbjg krb5kdc[2044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:19:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:19:22.968500 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45285
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:23.078440 2057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:23.078732 2057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:23.078775 2057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:23.082790 2057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:23.082854 2057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:23.082872 2057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:23.082888 2057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:23.082901 2057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:23.088330 2057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45285
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:23.089418 2057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:23.090462 2057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:23.099227 2065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:23.099237 2062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:23.099231 2063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:23.099465 2057 server_base.cc:1034] running on GCE node
I20260929 03:19:23.100205 2057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:23.101436 2057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:23.102603 2057 hybrid_clock.cc:648] HybridClock initialized: now 1790651963102585 us; error 34 us; skew 500 ppm
Sep 29 03:19:23 dist-test-slave-nbjg krb5kdc[2044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:23.105394 2057 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:19:23.106398 2057 webserver.cc:460] Webserver started at http://127.0.0.1:44691/ using document root <none> and password file <none>
I20260929 03:19:23.106971 2057 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:23.107038 2057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:23.107244 2057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:23.109347 2057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "537f3c2272b346559509c74ba84a9d7f"
format_stamp: "Formatted at 2026-09-29 03:19:23 on dist-test-slave-nbjg"
server_key: "00d30703f5b45dc43a37288949642c97"
server_key_iv: "43c027963a86939b0e79df9d2a7e4192"
server_key_version: "encryptionkey@0"
I20260929 03:19:23.109941 2057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "537f3c2272b346559509c74ba84a9d7f"
format_stamp: "Formatted at 2026-09-29 03:19:23 on dist-test-slave-nbjg"
server_key: "00d30703f5b45dc43a37288949642c97"
server_key_iv: "43c027963a86939b0e79df9d2a7e4192"
server_key_version: "encryptionkey@0"
I20260929 03:19:23.113859 2057 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260929 03:19:23.116487 2072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:23.117790 2057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:19:23.117913 2057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "537f3c2272b346559509c74ba84a9d7f"
format_stamp: "Formatted at 2026-09-29 03:19:23 on dist-test-slave-nbjg"
server_key: "00d30703f5b45dc43a37288949642c97"
server_key_iv: "43c027963a86939b0e79df9d2a7e4192"
server_key_version: "encryptionkey@0"
I20260929 03:19:23.118021 2057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:23.136526 2057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:23.139678 2057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:23.139871 2057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:23.146664 2057 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45285
I20260929 03:19:23.146703 2125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45285 every 8 connection(s)
I20260929 03:19:23.147630 2057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:23.150614 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:23.154466 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 2057
I20260929 03:19:23.154589 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:19:23.154850 14266 external_mini_cluster.cc:1428] Setting key 2af92d29df9e77ee101d02a3634e06bd
I20260929 03:19:23.156914 2126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 537f3c2272b346559509c74ba84a9d7f: Bootstrap starting.
I20260929 03:19:23.159845 2126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 537f3c2272b346559509c74ba84a9d7f: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:23.160772 2126 log.cc:824] T 00000000000000000000000000000000 P 537f3c2272b346559509c74ba84a9d7f: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:23.163223 2126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 537f3c2272b346559509c74ba84a9d7f: No bootstrap required, opened a new log
Sep 29 03:19:23 dist-test-slave-nbjg krb5kdc[2044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:19:23.167470 2126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 537f3c2272b346559509c74ba84a9d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537f3c2272b346559509c74ba84a9d7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45285 } }
I20260929 03:19:23.167773 2126 raft_consensus.cc:374] T 00000000000000000000000000000000 P 537f3c2272b346559509c74ba84a9d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:23.167830 2126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 537f3c2272b346559509c74ba84a9d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 537f3c2272b346559509c74ba84a9d7f, State: Initialized, Role: FOLLOWER
I20260929 03:19:23.168298 2126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 537f3c2272b346559509c74ba84a9d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537f3c2272b346559509c74ba84a9d7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45285 } }
I20260929 03:19:23.168426 2126 raft_consensus.cc:388] T 00000000000000000000000000000000 P 537f3c2272b346559509c74ba84a9d7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:23.168489 2126 raft_consensus.cc:482] T 00000000000000000000000000000000 P 537f3c2272b346559509c74ba84a9d7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:23.168589 2126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 537f3c2272b346559509c74ba84a9d7f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:23.169548 2126 raft_consensus.cc:504] T 00000000000000000000000000000000 P 537f3c2272b346559509c74ba84a9d7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537f3c2272b346559509c74ba84a9d7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45285 } }
I20260929 03:19:23.169937 2126 leader_election.cc:302] T 00000000000000000000000000000000 P 537f3c2272b346559509c74ba84a9d7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 537f3c2272b346559509c74ba84a9d7f; no voters:
I20260929 03:19:23.170307 2126 leader_election.cc:288] T 00000000000000000000000000000000 P 537f3c2272b346559509c74ba84a9d7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:19:23.170392 2131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 537f3c2272b346559509c74ba84a9d7f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:23.170563 2131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 537f3c2272b346559509c74ba84a9d7f [term 1 LEADER]: Becoming Leader. State: Replica: 537f3c2272b346559509c74ba84a9d7f, State: Running, Role: LEADER
I20260929 03:19:23.170878 2129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:23.156392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58301 (local address 127.0.0.1:45285)
0929 03:19:23.156849 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:23.156888 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:23.156933 (+ 45us) server_negotiation.cc:407] Connection header received
0929 03:19:23.157847 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:23.157873 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:23.158417 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:23.158814 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:23.159539 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:23.161060 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:23.162110 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:23.162845 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:23.163004 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:23.165409 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:23.165417 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:23.165437 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:23.165486 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:23.167435 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:23.168036 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:23.168042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:23.168046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:23.168208 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:23.168574 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:23.168579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:23.168580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:23.169028 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:23.169270 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:23.169615 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 03:19:23.169868 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":123,"threads_started":1}
I20260929 03:19:23.170882 2131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 537f3c2272b346559509c74ba84a9d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537f3c2272b346559509c74ba84a9d7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45285 } }
I20260929 03:19:23.171698 2126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 537f3c2272b346559509c74ba84a9d7f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:23.172649 2132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 537f3c2272b346559509c74ba84a9d7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 537f3c2272b346559509c74ba84a9d7f. Latest consensus state: current_term: 1 leader_uuid: "537f3c2272b346559509c74ba84a9d7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537f3c2272b346559509c74ba84a9d7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45285 } } }
I20260929 03:19:23.172793 2132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 537f3c2272b346559509c74ba84a9d7f [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:23.173106 2133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 537f3c2272b346559509c74ba84a9d7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "537f3c2272b346559509c74ba84a9d7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537f3c2272b346559509c74ba84a9d7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45285 } } }
I20260929 03:19:23.173197 2133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 537f3c2272b346559509c74ba84a9d7f [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:23.173511 2140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:23.174698 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:19:23.177008 2140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:23.183322 2140 catalog_manager.cc:1347] Generated new cluster ID: 7371d57e845643feb1fe1ce4c9bee478
I20260929 03:19:23.183395 2140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:23.186154 2129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:23.177134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40239 (local address 127.0.0.1:45285)
0929 03:19:23.177240 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:23.177245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:23.178499 (+ 1254us) server_negotiation.cc:407] Connection header received
0929 03:19:23.178623 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:23.178626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:23.178674 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:23.178758 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:23.179518 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:23.180033 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:23.180801 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:23.181082 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:23.181184 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:23.182691 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:23.182694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:23.182697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:23.182721 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:23.184329 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:23.184909 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:23.184913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:23.184915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:23.185008 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:23.185324 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:23.185327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:23.185329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:23.185511 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:23.185630 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:23.185899 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 03:19:23.185996 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:19:23.194779 2140 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:23.195874 2140 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:23.206526 2140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 537f3c2272b346559509c74ba84a9d7f: Generated new TSK 0
I20260929 03:19:23.207309 2140 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:19:23.218603 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 2057
W20260929 03:19:23.224148 2042 connection.cc:535] client connection to 127.0.0.1:45285 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45285): Connection reset by peer (error 104)
2026-09-29T03:19:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:19:23 dist-test-slave-nbjg krb5kdc[2157](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:19:23 dist-test-slave-nbjg krb5kdc[2157](info): set up 2 sockets
Sep 29 03:19:23 dist-test-slave-nbjg krb5kdc[2157](info): commencing operation
krb5kdc: starting...
W20260929 03:19:24.170445 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:19:24 dist-test-slave-nbjg krb5kdc[2157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:19:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:19:24.289958 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45427
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:24.399168 2171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:24.399472 2171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:24.399531 2171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:24.403656 2171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:24.403735 2171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:24.403760 2171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:24.403784 2171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:24.403812 2171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:24.409304 2171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45427
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:24.410467 2171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:24.411504 2171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:24.420260 2179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:24.420292 2177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:24.420277 2176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:24.420612 2171 server_base.cc:1034] running on GCE node
I20260929 03:19:24.421235 2171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:24.422472 2171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:24.423668 2171 hybrid_clock.cc:648] HybridClock initialized: now 1790651964423639 us; error 43 us; skew 500 ppm
Sep 29 03:19:24 dist-test-slave-nbjg krb5kdc[2157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:24.426527 2171 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:19:24.427456 2171 webserver.cc:460] Webserver started at http://127.0.0.1:38475/ using document root <none> and password file <none>
I20260929 03:19:24.428056 2171 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:24.428122 2171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:24.428329 2171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:24.430526 2171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "c9fe1e552a6f405781b5b5191a00267b"
format_stamp: "Formatted at 2026-09-29 03:19:24 on dist-test-slave-nbjg"
server_key: "44da792c955ec5dd3f41a6eecd466f76"
server_key_iv: "dd4358399f2c1409ca549437f662dfba"
server_key_version: "encryptionkey@0"
I20260929 03:19:24.431097 2171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "c9fe1e552a6f405781b5b5191a00267b"
format_stamp: "Formatted at 2026-09-29 03:19:24 on dist-test-slave-nbjg"
server_key: "44da792c955ec5dd3f41a6eecd466f76"
server_key_iv: "dd4358399f2c1409ca549437f662dfba"
server_key_version: "encryptionkey@0"
I20260929 03:19:24.434841 2171 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 03:19:24.437286 2186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:24.438607 2171 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 03:19:24.438722 2171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "c9fe1e552a6f405781b5b5191a00267b"
format_stamp: "Formatted at 2026-09-29 03:19:24 on dist-test-slave-nbjg"
server_key: "44da792c955ec5dd3f41a6eecd466f76"
server_key_iv: "dd4358399f2c1409ca549437f662dfba"
server_key_version: "encryptionkey@0"
I20260929 03:19:24.438818 2171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:24.449687 2171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:24.452783 2171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:24.452960 2171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:24.459712 2171 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45427
I20260929 03:19:24.459728 2239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45427 every 8 connection(s)
I20260929 03:19:24.460692 2171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:24.463500 2240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:24.466015 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 2171
I20260929 03:19:24.466161 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:19:24.466500 14266 external_mini_cluster.cc:1428] Setting key 6ef05306bf74eff7156b8cc4e76c455c
I20260929 03:19:24.469502 2240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9fe1e552a6f405781b5b5191a00267b: Bootstrap starting.
I20260929 03:19:24.472046 2240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9fe1e552a6f405781b5b5191a00267b: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:24.472822 2240 log.cc:824] T 00000000000000000000000000000000 P c9fe1e552a6f405781b5b5191a00267b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:24.474917 2240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9fe1e552a6f405781b5b5191a00267b: No bootstrap required, opened a new log
I20260929 03:19:24.475311 2243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:24.467861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52641 (local address 127.0.0.1:45427)
0929 03:19:24.468331 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:24.468347 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:24.468382 (+ 35us) server_negotiation.cc:407] Connection header received
0929 03:19:24.469000 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:24.469018 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:24.469373 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:24.469695 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:24.470021 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:24.471031 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:24.472187 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:24.472676 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:24.472788 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:24.473627 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:24.473632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:24.473646 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:24.474024 (+ 378us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:24.474084 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:24.474109 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:24.474252 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:24.474426 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 03:19:24.474568 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":180,"threads_started":1}
I20260929 03:19:24.478113 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:19:24.478574 2240 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9fe1e552a6f405781b5b5191a00267b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fe1e552a6f405781b5b5191a00267b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45427 } }
I20260929 03:19:24.478816 2240 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9fe1e552a6f405781b5b5191a00267b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:24.478859 2240 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9fe1e552a6f405781b5b5191a00267b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9fe1e552a6f405781b5b5191a00267b, State: Initialized, Role: FOLLOWER
I20260929 03:19:24.479298 2240 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9fe1e552a6f405781b5b5191a00267b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fe1e552a6f405781b5b5191a00267b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45427 } }
I20260929 03:19:24.479457 2240 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9fe1e552a6f405781b5b5191a00267b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:24.479524 2240 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9fe1e552a6f405781b5b5191a00267b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:24.479629 2240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9fe1e552a6f405781b5b5191a00267b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:24.480674 2240 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9fe1e552a6f405781b5b5191a00267b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fe1e552a6f405781b5b5191a00267b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45427 } }
I20260929 03:19:24.480991 2240 leader_election.cc:302] T 00000000000000000000000000000000 P c9fe1e552a6f405781b5b5191a00267b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9fe1e552a6f405781b5b5191a00267b; no voters:
I20260929 03:19:24.481267 2240 leader_election.cc:288] T 00000000000000000000000000000000 P c9fe1e552a6f405781b5b5191a00267b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:19:24.481354 2250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9fe1e552a6f405781b5b5191a00267b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:24.481519 2250 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9fe1e552a6f405781b5b5191a00267b [term 1 LEADER]: Becoming Leader. State: Replica: c9fe1e552a6f405781b5b5191a00267b, State: Running, Role: LEADER
I20260929 03:19:24.481815 2250 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9fe1e552a6f405781b5b5191a00267b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fe1e552a6f405781b5b5191a00267b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45427 } }
I20260929 03:19:24.482594 2240 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9fe1e552a6f405781b5b5191a00267b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:24.483016 2250 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9fe1e552a6f405781b5b5191a00267b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9fe1e552a6f405781b5b5191a00267b. Latest consensus state: current_term: 1 leader_uuid: "c9fe1e552a6f405781b5b5191a00267b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fe1e552a6f405781b5b5191a00267b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45427 } } }
I20260929 03:19:24.483134 2250 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9fe1e552a6f405781b5b5191a00267b [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:24.484246 2251 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9fe1e552a6f405781b5b5191a00267b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9fe1e552a6f405781b5b5191a00267b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fe1e552a6f405781b5b5191a00267b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45427 } } }
I20260929 03:19:24.484346 2251 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9fe1e552a6f405781b5b5191a00267b [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:24.485217 2243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:24.480145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32891 (local address 127.0.0.1:45427)
0929 03:19:24.480250 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:24.480255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:24.480418 (+ 163us) server_negotiation.cc:407] Connection header received
0929 03:19:24.480544 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:24.480547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:24.480590 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:24.480652 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:24.480907 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:24.481604 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:24.482547 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:24.482844 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:24.484471 (+ 1627us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:24.484611 (+ 140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:24.484615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:24.484617 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:24.484628 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:24.484649 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:24.484660 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:24.485036 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:24.485087 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 03:19:24.485120 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260929 03:19:24.487178 2265 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c9fe1e552a6f405781b5b5191a00267b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:19:24.487241 2265 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:19:24.487310 2254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:24.487958 2254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:24.492753 2254 catalog_manager.cc:1347] Generated new cluster ID: 5855a7accbc24a34899f17752947d2f3
I20260929 03:19:24.492811 2254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:24.506636 2254 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:24.507969 2254 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:24.523280 2254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9fe1e552a6f405781b5b5191a00267b: Generated new TSK 0
I20260929 03:19:24.524221 2254 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:19:24.533176 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 2171
W20260929 03:19:24.538341 2155 connection.cc:535] client connection to 127.0.0.1:45427 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45427): Connection reset by peer (error 104)
2026-09-29T03:19:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:19:24 dist-test-slave-nbjg krb5kdc[2271](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:19:24 dist-test-slave-nbjg krb5kdc[2271](info): set up 2 sockets
Sep 29 03:19:24 dist-test-slave-nbjg krb5kdc[2271](info): commencing operation
krb5kdc: starting...
W20260929 03:19:25.490617 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:19:25 dist-test-slave-nbjg krb5kdc[2271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:19:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:19:25.611589 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37533
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:25.723176 2284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:25.723445 2284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:25.723505 2284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:25.727562 2284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:25.727620 2284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:25.727645 2284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:25.727667 2284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:25.727697 2284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:25.732895 2284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37533
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:25.733940 2284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:25.735013 2284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:25.746297 2292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:25.746338 2284 server_base.cc:1034] running on GCE node
W20260929 03:19:25.746531 2290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:25.746299 2289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:25.747376 2284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:25.748621 2284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:25.749794 2284 hybrid_clock.cc:648] HybridClock initialized: now 1790651965749758 us; error 52 us; skew 500 ppm
Sep 29 03:19:25 dist-test-slave-nbjg krb5kdc[2271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:25.752673 2284 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:19:25.753562 2284 webserver.cc:460] Webserver started at http://127.0.0.1:46527/ using document root <none> and password file <none>
I20260929 03:19:25.754109 2284 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:25.754168 2284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:25.754428 2284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:25.757252 2284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "c902271c8d914bff984742cfff0aab54"
format_stamp: "Formatted at 2026-09-29 03:19:25 on dist-test-slave-nbjg"
server_key: "f5e828e077ed884d1a3bafafb7cee4a2"
server_key_iv: "c2fd86f6e3be7fb9b32d32b4c3c0e662"
server_key_version: "encryptionkey@0"
I20260929 03:19:25.758050 2284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "c902271c8d914bff984742cfff0aab54"
format_stamp: "Formatted at 2026-09-29 03:19:25 on dist-test-slave-nbjg"
server_key: "f5e828e077ed884d1a3bafafb7cee4a2"
server_key_iv: "c2fd86f6e3be7fb9b32d32b4c3c0e662"
server_key_version: "encryptionkey@0"
I20260929 03:19:25.763411 2284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 03:19:25.766619 2299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:25.768146 2284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:19:25.768292 2284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "c902271c8d914bff984742cfff0aab54"
format_stamp: "Formatted at 2026-09-29 03:19:25 on dist-test-slave-nbjg"
server_key: "f5e828e077ed884d1a3bafafb7cee4a2"
server_key_iv: "c2fd86f6e3be7fb9b32d32b4c3c0e662"
server_key_version: "encryptionkey@0"
I20260929 03:19:25.768420 2284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:25.773494 2284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:25.777889 2284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:25.778119 2284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:25.788136 2284 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37533
I20260929 03:19:25.788164 2351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37533 every 8 connection(s)
I20260929 03:19:25.789620 2284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:25.792825 2352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:25.798005 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 2284
I20260929 03:19:25.798151 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:19:25.798434 14266 external_mini_cluster.cc:1428] Setting key dfc202ca5dc7a267301185859de4ce88
I20260929 03:19:25.799494 2352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c902271c8d914bff984742cfff0aab54: Bootstrap starting.
I20260929 03:19:25.802392 2352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c902271c8d914bff984742cfff0aab54: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:25.803469 2352 log.cc:824] T 00000000000000000000000000000000 P c902271c8d914bff984742cfff0aab54: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:25.804787 2355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:25.800004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55739 (local address 127.0.0.1:37533)
0929 03:19:25.800501 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:25.800519 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:25.800564 (+ 45us) server_negotiation.cc:407] Connection header received
0929 03:19:25.801470 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:25.801496 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:25.801894 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:25.802347 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:25.803338 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:25.803344 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:25.803362 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:25.803375 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:25.803454 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:25.803484 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:25.803602 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:25.803730 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 03:19:25.803907 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":139,"threads_started":1}
I20260929 03:19:25.805823 2352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c902271c8d914bff984742cfff0aab54: No bootstrap required, opened a new log
I20260929 03:19:25.808040 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:19:25.809756 2352 raft_consensus.cc:348] T 00000000000000000000000000000000 P c902271c8d914bff984742cfff0aab54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c902271c8d914bff984742cfff0aab54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37533 } }
I20260929 03:19:25.810070 2352 raft_consensus.cc:374] T 00000000000000000000000000000000 P c902271c8d914bff984742cfff0aab54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:25.810184 2352 raft_consensus.cc:729] T 00000000000000000000000000000000 P c902271c8d914bff984742cfff0aab54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902271c8d914bff984742cfff0aab54, State: Initialized, Role: FOLLOWER
I20260929 03:19:25.810742 2352 consensus_queue.cc:260] T 00000000000000000000000000000000 P c902271c8d914bff984742cfff0aab54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c902271c8d914bff984742cfff0aab54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37533 } }
I20260929 03:19:25.810937 2352 raft_consensus.cc:388] T 00000000000000000000000000000000 P c902271c8d914bff984742cfff0aab54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:25.811010 2352 raft_consensus.cc:482] T 00000000000000000000000000000000 P c902271c8d914bff984742cfff0aab54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:25.811111 2352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c902271c8d914bff984742cfff0aab54 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:25.811951 2355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:25.810148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49615 (local address 127.0.0.1:37533)
0929 03:19:25.810289 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:25.810293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:25.810577 (+ 284us) server_negotiation.cc:407] Connection header received
0929 03:19:25.810680 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:25.810683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:25.810722 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:25.810807 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:25.811607 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:25.811611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:25.811613 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:25.811626 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:25.811656 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:25.811668 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:25.811746 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:25.811799 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 03:19:25.811835 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:19:25.812321 2352 raft_consensus.cc:504] T 00000000000000000000000000000000 P c902271c8d914bff984742cfff0aab54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c902271c8d914bff984742cfff0aab54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37533 } }
I20260929 03:19:25.812667 2352 leader_election.cc:302] T 00000000000000000000000000000000 P c902271c8d914bff984742cfff0aab54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c902271c8d914bff984742cfff0aab54; no voters:
I20260929 03:19:25.812999 2352 leader_election.cc:288] T 00000000000000000000000000000000 P c902271c8d914bff984742cfff0aab54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:19:25.813144 2362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c902271c8d914bff984742cfff0aab54 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:25.813359 2362 raft_consensus.cc:686] T 00000000000000000000000000000000 P c902271c8d914bff984742cfff0aab54 [term 1 LEADER]: Becoming Leader. State: Replica: c902271c8d914bff984742cfff0aab54, State: Running, Role: LEADER
I20260929 03:19:25.813730 2362 consensus_queue.cc:237] T 00000000000000000000000000000000 P c902271c8d914bff984742cfff0aab54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c902271c8d914bff984742cfff0aab54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37533 } }
I20260929 03:19:25.814312 2352 sys_catalog.cc:564] T 00000000000000000000000000000000 P c902271c8d914bff984742cfff0aab54 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:25.815415 2364 sys_catalog.cc:455] T 00000000000000000000000000000000 P c902271c8d914bff984742cfff0aab54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c902271c8d914bff984742cfff0aab54. Latest consensus state: current_term: 1 leader_uuid: "c902271c8d914bff984742cfff0aab54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c902271c8d914bff984742cfff0aab54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37533 } } }
I20260929 03:19:25.815397 2363 sys_catalog.cc:455] T 00000000000000000000000000000000 P c902271c8d914bff984742cfff0aab54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c902271c8d914bff984742cfff0aab54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c902271c8d914bff984742cfff0aab54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37533 } } }
I20260929 03:19:25.815541 2364 sys_catalog.cc:458] T 00000000000000000000000000000000 P c902271c8d914bff984742cfff0aab54 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:25.815541 2363 sys_catalog.cc:458] T 00000000000000000000000000000000 P c902271c8d914bff984742cfff0aab54 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:25.815912 2371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:25.819307 2371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:25.825560 2371 catalog_manager.cc:1347] Generated new cluster ID: 545d20efe6f543f084a3ecbc9325fd4f
I20260929 03:19:25.825629 2371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:25.851399 2371 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:25.851732 2371 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:25.863435 2371 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c902271c8d914bff984742cfff0aab54: Generated new TSK 0
I20260929 03:19:25.864274 2371 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:19:25.874778 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 2284
2026-09-29T03:19:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:19:25 dist-test-slave-nbjg krb5kdc[2383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:19:25 dist-test-slave-nbjg krb5kdc[2383](info): set up 2 sockets
Sep 29 03:19:25 dist-test-slave-nbjg krb5kdc[2383](info): commencing operation
krb5kdc: starting...
W20260929 03:19:26.834344 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:19:26 dist-test-slave-nbjg krb5kdc[2383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:19:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:19:26.953243 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42405
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:27.063431 2396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:27.063751 2396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:27.063795 2396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:27.067859 2396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:27.067934 2396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:27.067952 2396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:27.067968 2396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:27.067982 2396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:27.073441 2396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:27.074572 2396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:27.075594 2396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:27.084074 2401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:27.084086 2404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:27.084086 2402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:27.084482 2396 server_base.cc:1034] running on GCE node
I20260929 03:19:27.085024 2396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:27.086135 2396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:27.087296 2396 hybrid_clock.cc:648] HybridClock initialized: now 1790651967087258 us; error 43 us; skew 500 ppm
Sep 29 03:19:27 dist-test-slave-nbjg krb5kdc[2383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:27.094061 2396 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:19:27.095010 2396 webserver.cc:460] Webserver started at http://127.0.0.1:34255/ using document root <none> and password file <none>
I20260929 03:19:27.095551 2396 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:27.095611 2396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:27.095795 2396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:27.097823 2396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "8afed0f3b5c94871aaf3d9c04bbb604d"
format_stamp: "Formatted at 2026-09-29 03:19:27 on dist-test-slave-nbjg"
server_key: "53e54c57c0bafb7184f0e69c6544348e"
server_key_iv: "ff3d601641255ce7e170eef14e6a5c74"
server_key_version: "encryptionkey@0"
I20260929 03:19:27.098377 2396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "8afed0f3b5c94871aaf3d9c04bbb604d"
format_stamp: "Formatted at 2026-09-29 03:19:27 on dist-test-slave-nbjg"
server_key: "53e54c57c0bafb7184f0e69c6544348e"
server_key_iv: "ff3d601641255ce7e170eef14e6a5c74"
server_key_version: "encryptionkey@0"
I20260929 03:19:27.101989 2396 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 03:19:27.104295 2411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:27.105598 2396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:19:27.105731 2396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "8afed0f3b5c94871aaf3d9c04bbb604d"
format_stamp: "Formatted at 2026-09-29 03:19:27 on dist-test-slave-nbjg"
server_key: "53e54c57c0bafb7184f0e69c6544348e"
server_key_iv: "ff3d601641255ce7e170eef14e6a5c74"
server_key_version: "encryptionkey@0"
I20260929 03:19:27.105826 2396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:27.119638 2396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:27.122804 2396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:27.122994 2396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:27.129983 2396 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42405
I20260929 03:19:27.129998 2463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42405 every 8 connection(s)
I20260929 03:19:27.130972 2396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:27.133800 2464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:27.139309 2464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8afed0f3b5c94871aaf3d9c04bbb604d: Bootstrap starting.
I20260929 03:19:27.139343 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 2396
I20260929 03:19:27.139465 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:19:27.139704 14266 external_mini_cluster.cc:1428] Setting key 79cf667dea90d15baedaccb64f6e1ea4
I20260929 03:19:27.141746 2464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8afed0f3b5c94871aaf3d9c04bbb604d: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:27.142479 2464 log.cc:824] T 00000000000000000000000000000000 P 8afed0f3b5c94871aaf3d9c04bbb604d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:27.144375 2464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8afed0f3b5c94871aaf3d9c04bbb604d: No bootstrap required, opened a new log
I20260929 03:19:27.148051 2464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8afed0f3b5c94871aaf3d9c04bbb604d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8afed0f3b5c94871aaf3d9c04bbb604d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42405 } }
Sep 29 03:19:27 dist-test-slave-nbjg krb5kdc[2383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:19:27.148274 2464 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8afed0f3b5c94871aaf3d9c04bbb604d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:27.148346 2464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8afed0f3b5c94871aaf3d9c04bbb604d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8afed0f3b5c94871aaf3d9c04bbb604d, State: Initialized, Role: FOLLOWER
I20260929 03:19:27.148780 2464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8afed0f3b5c94871aaf3d9c04bbb604d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8afed0f3b5c94871aaf3d9c04bbb604d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42405 } }
I20260929 03:19:27.148939 2464 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8afed0f3b5c94871aaf3d9c04bbb604d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:27.149013 2464 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8afed0f3b5c94871aaf3d9c04bbb604d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:27.149116 2464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8afed0f3b5c94871aaf3d9c04bbb604d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:27.150168 2464 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8afed0f3b5c94871aaf3d9c04bbb604d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8afed0f3b5c94871aaf3d9c04bbb604d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42405 } }
I20260929 03:19:27.150563 2464 leader_election.cc:302] T 00000000000000000000000000000000 P 8afed0f3b5c94871aaf3d9c04bbb604d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8afed0f3b5c94871aaf3d9c04bbb604d; no voters:
I20260929 03:19:27.150911 2464 leader_election.cc:288] T 00000000000000000000000000000000 P 8afed0f3b5c94871aaf3d9c04bbb604d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:19:27.151002 2469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8afed0f3b5c94871aaf3d9c04bbb604d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:27.151223 2469 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8afed0f3b5c94871aaf3d9c04bbb604d [term 1 LEADER]: Becoming Leader. State: Replica: 8afed0f3b5c94871aaf3d9c04bbb604d, State: Running, Role: LEADER
I20260929 03:19:27.151611 2469 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8afed0f3b5c94871aaf3d9c04bbb604d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8afed0f3b5c94871aaf3d9c04bbb604d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42405 } }
I20260929 03:19:27.152251 2464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8afed0f3b5c94871aaf3d9c04bbb604d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:27.153301 2471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8afed0f3b5c94871aaf3d9c04bbb604d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8afed0f3b5c94871aaf3d9c04bbb604d. Latest consensus state: current_term: 1 leader_uuid: "8afed0f3b5c94871aaf3d9c04bbb604d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8afed0f3b5c94871aaf3d9c04bbb604d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42405 } } }
I20260929 03:19:27.153446 2471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8afed0f3b5c94871aaf3d9c04bbb604d [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:27.153920 2478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:27.154892 2467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:27.141124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49417 (local address 127.0.0.1:42405)
0929 03:19:27.141546 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:27.141561 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:27.141596 (+ 35us) server_negotiation.cc:407] Connection header received
0929 03:19:27.142279 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:27.142300 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:27.142683 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:27.143003 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:27.143783 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:27.144788 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:27.145710 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:27.146226 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:27.146379 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:27.148843 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:27.148850 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:27.148868 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:27.148900 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:27.150871 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:27.151380 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:27.151386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:27.151392 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:27.151550 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:27.151859 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:27.151862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:27.151864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:27.152420 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:27.152590 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:27.153592 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0929 03:19:27.153909 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":107,"threads_started":1}
I20260929 03:19:27.153301 2470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8afed0f3b5c94871aaf3d9c04bbb604d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8afed0f3b5c94871aaf3d9c04bbb604d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8afed0f3b5c94871aaf3d9c04bbb604d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42405 } } }
I20260929 03:19:27.155516 2470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8afed0f3b5c94871aaf3d9c04bbb604d [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:27.157493 2478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:27.158497 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:19:27.163477 2478 catalog_manager.cc:1347] Generated new cluster ID: 5d35018cb64c4269a21081ff72f73e14
I20260929 03:19:27.163544 2478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:27.169106 2467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:27.160551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40895 (local address 127.0.0.1:42405)
0929 03:19:27.160654 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:27.160658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:27.160965 (+ 307us) server_negotiation.cc:407] Connection header received
0929 03:19:27.161134 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:27.161137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:27.161182 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:27.161243 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:27.162003 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:27.162780 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:27.163941 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:27.164308 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:27.164404 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:27.165946 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:27.165950 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:27.165952 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:27.165976 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:27.167554 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:27.167998 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:27.168002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:27.168004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:27.168086 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:27.168336 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:27.168338 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:27.168340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:27.168518 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:27.168642 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:27.168841 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 03:19:27.168944 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":35}
I20260929 03:19:27.186751 2478 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:27.188105 2478 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:27.197640 2478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8afed0f3b5c94871aaf3d9c04bbb604d: Generated new TSK 0
I20260929 03:19:27.198558 2478 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:19:27.205801 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 2396
2026-09-29T03:19:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:19:27 dist-test-slave-nbjg krb5kdc[2495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:19:27 dist-test-slave-nbjg krb5kdc[2495](info): set up 2 sockets
Sep 29 03:19:27 dist-test-slave-nbjg krb5kdc[2495](info): commencing operation
krb5kdc: starting...
W20260929 03:19:28.158349 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:19:28 dist-test-slave-nbjg krb5kdc[2495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:19:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:19:28.275128 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41553
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:28.383183 2508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:28.383447 2508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:28.383505 2508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:28.387465 2508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:28.387524 2508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:28.387549 2508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:28.387570 2508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:28.387590 2508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:28.392764 2508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:28.393786 2508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:28.394838 2508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:28.403255 2514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:28.403262 2513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:28.403432 2516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:28.403833 2508 server_base.cc:1034] running on GCE node
I20260929 03:19:28.404409 2508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:28.405566 2508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:28.406719 2508 hybrid_clock.cc:648] HybridClock initialized: now 1790651968406694 us; error 40 us; skew 500 ppm
Sep 29 03:19:28 dist-test-slave-nbjg krb5kdc[2495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:28.409413 2508 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:19:28.410370 2508 webserver.cc:460] Webserver started at http://127.0.0.1:37607/ using document root <none> and password file <none>
I20260929 03:19:28.410943 2508 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:28.411010 2508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:28.411212 2508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:28.413259 2508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "dbdd398e7e4a4284a4b9546ad980ee2a"
format_stamp: "Formatted at 2026-09-29 03:19:28 on dist-test-slave-nbjg"
server_key: "642fc132c3ba3602ca107f2ee3bc5f01"
server_key_iv: "0d3e5ba8de9f1e9863f7212358fdb97b"
server_key_version: "encryptionkey@0"
I20260929 03:19:28.413794 2508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "dbdd398e7e4a4284a4b9546ad980ee2a"
format_stamp: "Formatted at 2026-09-29 03:19:28 on dist-test-slave-nbjg"
server_key: "642fc132c3ba3602ca107f2ee3bc5f01"
server_key_iv: "0d3e5ba8de9f1e9863f7212358fdb97b"
server_key_version: "encryptionkey@0"
I20260929 03:19:28.417680 2508 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:19:28.420097 2523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:28.421312 2508 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:19:28.421427 2508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "dbdd398e7e4a4284a4b9546ad980ee2a"
format_stamp: "Formatted at 2026-09-29 03:19:28 on dist-test-slave-nbjg"
server_key: "642fc132c3ba3602ca107f2ee3bc5f01"
server_key_iv: "0d3e5ba8de9f1e9863f7212358fdb97b"
server_key_version: "encryptionkey@0"
I20260929 03:19:28.421519 2508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:28.441583 2508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:28.444728 2508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:28.444906 2508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:28.451840 2508 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41553
I20260929 03:19:28.451869 2575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41553 every 8 connection(s)
I20260929 03:19:28.452777 2508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:28.455606 2576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:28.461051 2576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbdd398e7e4a4284a4b9546ad980ee2a: Bootstrap starting.
I20260929 03:19:28.461212 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 2508
I20260929 03:19:28.461346 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:19:28.461622 14266 external_mini_cluster.cc:1428] Setting key 4e05eb18e9901c28e03a5504c996752b
I20260929 03:19:28.463687 2576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbdd398e7e4a4284a4b9546ad980ee2a: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:28.464385 2576 log.cc:824] T 00000000000000000000000000000000 P dbdd398e7e4a4284a4b9546ad980ee2a: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:28.466289 2576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbdd398e7e4a4284a4b9546ad980ee2a: No bootstrap required, opened a new log
I20260929 03:19:28.470188 2576 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbdd398e7e4a4284a4b9546ad980ee2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbdd398e7e4a4284a4b9546ad980ee2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41553 } }
I20260929 03:19:28.470378 2579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:28.462927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58299 (local address 127.0.0.1:41553)
0929 03:19:28.463332 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:28.463348 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:28.463383 (+ 35us) server_negotiation.cc:407] Connection header received
0929 03:19:28.464048 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:28.464067 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:28.464419 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:28.464735 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:28.465164 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:28.466138 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:28.467361 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:28.467836 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:28.467956 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:28.468738 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:28.468743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:28.468755 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:28.469119 (+ 364us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:28.469170 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:28.469191 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:28.469263 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:28.469398 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 03:19:28.469592 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":113,"threads_started":1}
I20260929 03:19:28.470427 2576 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbdd398e7e4a4284a4b9546ad980ee2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:28.470465 2576 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbdd398e7e4a4284a4b9546ad980ee2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbdd398e7e4a4284a4b9546ad980ee2a, State: Initialized, Role: FOLLOWER
I20260929 03:19:28.470860 2576 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbdd398e7e4a4284a4b9546ad980ee2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbdd398e7e4a4284a4b9546ad980ee2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41553 } }
I20260929 03:19:28.470976 2576 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbdd398e7e4a4284a4b9546ad980ee2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:28.471042 2576 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbdd398e7e4a4284a4b9546ad980ee2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:28.471132 2576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbdd398e7e4a4284a4b9546ad980ee2a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:28.471939 2576 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbdd398e7e4a4284a4b9546ad980ee2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbdd398e7e4a4284a4b9546ad980ee2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41553 } }
I20260929 03:19:28.472275 2576 leader_election.cc:302] T 00000000000000000000000000000000 P dbdd398e7e4a4284a4b9546ad980ee2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbdd398e7e4a4284a4b9546ad980ee2a; no voters:
I20260929 03:19:28.472594 2576 leader_election.cc:288] T 00000000000000000000000000000000 P dbdd398e7e4a4284a4b9546ad980ee2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:19:28.472657 2581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbdd398e7e4a4284a4b9546ad980ee2a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:28.472854 2581 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbdd398e7e4a4284a4b9546ad980ee2a [term 1 LEADER]: Becoming Leader. State: Replica: dbdd398e7e4a4284a4b9546ad980ee2a, State: Running, Role: LEADER
I20260929 03:19:28.473074 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:19:28.473208 2581 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbdd398e7e4a4284a4b9546ad980ee2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbdd398e7e4a4284a4b9546ad980ee2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41553 } }
I20260929 03:19:28.473856 2576 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbdd398e7e4a4284a4b9546ad980ee2a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:28.474606 2582 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbdd398e7e4a4284a4b9546ad980ee2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbdd398e7e4a4284a4b9546ad980ee2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbdd398e7e4a4284a4b9546ad980ee2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41553 } } }
I20260929 03:19:28.474720 2582 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbdd398e7e4a4284a4b9546ad980ee2a [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:28.474982 2583 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbdd398e7e4a4284a4b9546ad980ee2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbdd398e7e4a4284a4b9546ad980ee2a. Latest consensus state: current_term: 1 leader_uuid: "dbdd398e7e4a4284a4b9546ad980ee2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbdd398e7e4a4284a4b9546ad980ee2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41553 } } }
I20260929 03:19:28.475066 2583 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbdd398e7e4a4284a4b9546ad980ee2a [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:28.475364 2594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:28.479233 2594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:28.481067 2579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:28.476415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36383 (local address 127.0.0.1:41553)
0929 03:19:28.476511 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:28.476515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:28.476726 (+ 211us) server_negotiation.cc:407] Connection header received
0929 03:19:28.476822 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:28.476825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:28.476870 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:28.476971 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:28.477295 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:28.478035 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:28.479326 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:28.479657 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:28.479769 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:28.480640 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:28.480643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:28.480646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:28.480660 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:28.480690 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:28.480702 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:28.480774 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:28.480873 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 03:19:28.480931 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:19:28.485756 2594 catalog_manager.cc:1347] Generated new cluster ID: 4b46bc9f44e24f7baf91571f0915216e
I20260929 03:19:28.485826 2594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:28.498845 2594 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:28.500252 2594 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:28.513360 2594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbdd398e7e4a4284a4b9546ad980ee2a: Generated new TSK 0
I20260929 03:19:28.514362 2594 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:19:28.524250 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 2508
2026-09-29T03:19:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:19:28 dist-test-slave-nbjg krb5kdc[2607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:19:28 dist-test-slave-nbjg krb5kdc[2607](info): set up 2 sockets
Sep 29 03:19:28 dist-test-slave-nbjg krb5kdc[2607](info): commencing operation
krb5kdc: starting...
W20260929 03:19:29.482427 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:19:29 dist-test-slave-nbjg krb5kdc[2607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:19:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:19:29.599467 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42173
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:29.711182 2620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:29.711429 2620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:29.711470 2620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:29.715420 2620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:29.715478 2620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:29.715495 2620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:29.715510 2620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:29.715523 2620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:29.720784 2620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:29.721807 2620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:29.722837 2620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:29.731046 2625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:29.731040 2626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:29.731211 2620 server_base.cc:1034] running on GCE node
W20260929 03:19:29.731040 2628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:29.731832 2620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:29.732887 2620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:29.734058 2620 hybrid_clock.cc:648] HybridClock initialized: now 1790651969734034 us; error 34 us; skew 500 ppm
Sep 29 03:19:29 dist-test-slave-nbjg krb5kdc[2607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:29.736680 2620 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:19:29.737646 2620 webserver.cc:460] Webserver started at http://127.0.0.1:43449/ using document root <none> and password file <none>
I20260929 03:19:29.738225 2620 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:29.738302 2620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:29.738507 2620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:29.740594 2620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "f33af78f6dac4e109d10b212abe985c2"
format_stamp: "Formatted at 2026-09-29 03:19:29 on dist-test-slave-nbjg"
server_key: "c45c8894dc8f97b6736c12080848b28c"
server_key_iv: "e150be29e4dd6a441ae0bedf36b5a045"
server_key_version: "encryptionkey@0"
I20260929 03:19:29.741183 2620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "f33af78f6dac4e109d10b212abe985c2"
format_stamp: "Formatted at 2026-09-29 03:19:29 on dist-test-slave-nbjg"
server_key: "c45c8894dc8f97b6736c12080848b28c"
server_key_iv: "e150be29e4dd6a441ae0bedf36b5a045"
server_key_version: "encryptionkey@0"
I20260929 03:19:29.745110 2620 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:19:29.747689 2635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:29.749033 2620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:19:29.749159 2620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "f33af78f6dac4e109d10b212abe985c2"
format_stamp: "Formatted at 2026-09-29 03:19:29 on dist-test-slave-nbjg"
server_key: "c45c8894dc8f97b6736c12080848b28c"
server_key_iv: "e150be29e4dd6a441ae0bedf36b5a045"
server_key_version: "encryptionkey@0"
I20260929 03:19:29.749253 2620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:29.753916 2620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:29.757155 2620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:29.757305 2620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:29.764115 2620 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42173
I20260929 03:19:29.764120 2687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42173 every 8 connection(s)
I20260929 03:19:29.765079 2620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:29.765480 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 2620
I20260929 03:19:29.765611 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:19:29.765870 14266 external_mini_cluster.cc:1428] Setting key ee76a2bef6a5bd9c59463822226298a6
I20260929 03:19:29.768306 2688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:29.771282 2691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:29.767361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41293 (local address 127.0.0.1:42173)
0929 03:19:29.767805 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:29.767821 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:29.767858 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:19:29.768598 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:29.768617 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:29.768917 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:29.769225 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:29.770060 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:29.770065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:29.770078 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:29.770088 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:29.770142 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:29.770162 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:29.770299 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:29.770389 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 03:19:29.770537 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":121,"threads_started":1}
I20260929 03:19:29.774214 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:19:29.774992 2688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f33af78f6dac4e109d10b212abe985c2: Bootstrap starting.
I20260929 03:19:29.777935 2688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f33af78f6dac4e109d10b212abe985c2: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:29.778270 2691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:29.776567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60827 (local address 127.0.0.1:42173)
0929 03:19:29.776701 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:29.776705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:29.776836 (+ 131us) server_negotiation.cc:407] Connection header received
0929 03:19:29.776977 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:29.776979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:29.777010 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:29.777074 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:29.777833 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:29.777836 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:29.777839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:29.777850 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:29.777881 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:29.777892 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:29.777973 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:29.778082 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 03:19:29.778124 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 03:19:29.778910 2688 log.cc:824] T 00000000000000000000000000000000 P f33af78f6dac4e109d10b212abe985c2: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:29.781018 2688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f33af78f6dac4e109d10b212abe985c2: No bootstrap required, opened a new log
I20260929 03:19:29.784674 2688 raft_consensus.cc:348] T 00000000000000000000000000000000 P f33af78f6dac4e109d10b212abe985c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33af78f6dac4e109d10b212abe985c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42173 } }
I20260929 03:19:29.784914 2688 raft_consensus.cc:374] T 00000000000000000000000000000000 P f33af78f6dac4e109d10b212abe985c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:29.784957 2688 raft_consensus.cc:729] T 00000000000000000000000000000000 P f33af78f6dac4e109d10b212abe985c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f33af78f6dac4e109d10b212abe985c2, State: Initialized, Role: FOLLOWER
I20260929 03:19:29.785454 2688 consensus_queue.cc:260] T 00000000000000000000000000000000 P f33af78f6dac4e109d10b212abe985c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33af78f6dac4e109d10b212abe985c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42173 } }
I20260929 03:19:29.785589 2688 raft_consensus.cc:388] T 00000000000000000000000000000000 P f33af78f6dac4e109d10b212abe985c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:29.785656 2688 raft_consensus.cc:482] T 00000000000000000000000000000000 P f33af78f6dac4e109d10b212abe985c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:29.785753 2688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f33af78f6dac4e109d10b212abe985c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:29.786880 2688 raft_consensus.cc:504] T 00000000000000000000000000000000 P f33af78f6dac4e109d10b212abe985c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33af78f6dac4e109d10b212abe985c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42173 } }
I20260929 03:19:29.787257 2688 leader_election.cc:302] T 00000000000000000000000000000000 P f33af78f6dac4e109d10b212abe985c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f33af78f6dac4e109d10b212abe985c2; no voters:
I20260929 03:19:29.787588 2688 leader_election.cc:288] T 00000000000000000000000000000000 P f33af78f6dac4e109d10b212abe985c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:19:29.787695 2698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f33af78f6dac4e109d10b212abe985c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:29.787896 2698 raft_consensus.cc:686] T 00000000000000000000000000000000 P f33af78f6dac4e109d10b212abe985c2 [term 1 LEADER]: Becoming Leader. State: Replica: f33af78f6dac4e109d10b212abe985c2, State: Running, Role: LEADER
I20260929 03:19:29.788228 2698 consensus_queue.cc:237] T 00000000000000000000000000000000 P f33af78f6dac4e109d10b212abe985c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33af78f6dac4e109d10b212abe985c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42173 } }
I20260929 03:19:29.788780 2688 sys_catalog.cc:564] T 00000000000000000000000000000000 P f33af78f6dac4e109d10b212abe985c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:29.789806 2699 sys_catalog.cc:455] T 00000000000000000000000000000000 P f33af78f6dac4e109d10b212abe985c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f33af78f6dac4e109d10b212abe985c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33af78f6dac4e109d10b212abe985c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42173 } } }
I20260929 03:19:29.789937 2699 sys_catalog.cc:458] T 00000000000000000000000000000000 P f33af78f6dac4e109d10b212abe985c2 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:29.790124 2700 sys_catalog.cc:455] T 00000000000000000000000000000000 P f33af78f6dac4e109d10b212abe985c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f33af78f6dac4e109d10b212abe985c2. Latest consensus state: current_term: 1 leader_uuid: "f33af78f6dac4e109d10b212abe985c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33af78f6dac4e109d10b212abe985c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42173 } } }
I20260929 03:19:29.790301 2700 sys_catalog.cc:458] T 00000000000000000000000000000000 P f33af78f6dac4e109d10b212abe985c2 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:29.790462 2707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:29.793728 2707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:29.799009 2707 catalog_manager.cc:1347] Generated new cluster ID: cf4738d77ac946748dc7b82386d7df06
I20260929 03:19:29.799069 2707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:29.821753 2707 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:29.822134 2707 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:29.833878 2707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f33af78f6dac4e109d10b212abe985c2: Generated new TSK 0
I20260929 03:19:29.834836 2707 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:19:29.842902 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 2620
2026-09-29T03:19:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:19:29 dist-test-slave-nbjg krb5kdc[2719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:19:29 dist-test-slave-nbjg krb5kdc[2719](info): set up 2 sockets
Sep 29 03:19:29 dist-test-slave-nbjg krb5kdc[2719](info): commencing operation
krb5kdc: starting...
W20260929 03:19:30.798538 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:19:30 dist-test-slave-nbjg krb5kdc[2719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:19:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:19:30.916860 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32783
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:31.026472 2732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:31.026754 2732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:31.026814 2732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:31.030864 2732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:31.030929 2732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:31.030956 2732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:31.030977 2732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:31.030997 2732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:31.036427 2732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:31.037591 2732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:31.038733 2732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:31.048058 2739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:31.048108 2732 server_base.cc:1034] running on GCE node
W20260929 03:19:31.048048 2741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:31.048062 2738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:31.048802 2732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:31.049985 2732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:31.051155 2732 hybrid_clock.cc:648] HybridClock initialized: now 1790651971051140 us; error 51 us; skew 500 ppm
Sep 29 03:19:31 dist-test-slave-nbjg krb5kdc[2719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:31.053817 2732 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:19:31.054736 2732 webserver.cc:460] Webserver started at http://127.0.0.1:40365/ using document root <none> and password file <none>
I20260929 03:19:31.055256 2732 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:31.055310 2732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:31.055475 2732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:31.057474 2732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "5288a8df8640457184f756a9c457f819"
format_stamp: "Formatted at 2026-09-29 03:19:31 on dist-test-slave-nbjg"
server_key: "a6dfba0336b99daaf7e5f1fcb3781d7d"
server_key_iv: "5ef00c56d31b6544d42b59c45dd8a20d"
server_key_version: "encryptionkey@0"
I20260929 03:19:31.057969 2732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "5288a8df8640457184f756a9c457f819"
format_stamp: "Formatted at 2026-09-29 03:19:31 on dist-test-slave-nbjg"
server_key: "a6dfba0336b99daaf7e5f1fcb3781d7d"
server_key_iv: "5ef00c56d31b6544d42b59c45dd8a20d"
server_key_version: "encryptionkey@0"
I20260929 03:19:31.061589 2732 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 03:19:31.063995 2748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:31.065208 2732 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 03:19:31.065310 2732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "5288a8df8640457184f756a9c457f819"
format_stamp: "Formatted at 2026-09-29 03:19:31 on dist-test-slave-nbjg"
server_key: "a6dfba0336b99daaf7e5f1fcb3781d7d"
server_key_iv: "5ef00c56d31b6544d42b59c45dd8a20d"
server_key_version: "encryptionkey@0"
I20260929 03:19:31.065394 2732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:31.093405 2732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:31.096521 2732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:31.096679 2732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:31.103520 2732 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32783
I20260929 03:19:31.103600 2800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32783 every 8 connection(s)
I20260929 03:19:31.104504 2732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:31.107394 2801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:31.113049 2801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5288a8df8640457184f756a9c457f819: Bootstrap starting.
I20260929 03:19:31.113179 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 2732
I20260929 03:19:31.113328 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:19:31.113580 14266 external_mini_cluster.cc:1428] Setting key 8cf590291c93b780ddcfdbd699523757
I20260929 03:19:31.115693 2801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5288a8df8640457184f756a9c457f819: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:31.116425 2801 log.cc:824] T 00000000000000000000000000000000 P 5288a8df8640457184f756a9c457f819: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:31.118409 2801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5288a8df8640457184f756a9c457f819: No bootstrap required, opened a new log
Sep 29 03:19:31 dist-test-slave-nbjg krb5kdc[2719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:19:31.121950 2801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5288a8df8640457184f756a9c457f819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5288a8df8640457184f756a9c457f819" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32783 } }
I20260929 03:19:31.122268 2801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5288a8df8640457184f756a9c457f819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:31.122332 2801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5288a8df8640457184f756a9c457f819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5288a8df8640457184f756a9c457f819, State: Initialized, Role: FOLLOWER
I20260929 03:19:31.122764 2801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5288a8df8640457184f756a9c457f819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5288a8df8640457184f756a9c457f819" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32783 } }
I20260929 03:19:31.122893 2801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5288a8df8640457184f756a9c457f819 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:31.122957 2801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5288a8df8640457184f756a9c457f819 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:31.123054 2801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5288a8df8640457184f756a9c457f819 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:31.123907 2801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5288a8df8640457184f756a9c457f819 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5288a8df8640457184f756a9c457f819" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32783 } }
I20260929 03:19:31.124238 2801 leader_election.cc:302] T 00000000000000000000000000000000 P 5288a8df8640457184f756a9c457f819 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5288a8df8640457184f756a9c457f819; no voters:
I20260929 03:19:31.124527 2801 leader_election.cc:288] T 00000000000000000000000000000000 P 5288a8df8640457184f756a9c457f819 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:19:31.124617 2806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5288a8df8640457184f756a9c457f819 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:31.124816 2806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5288a8df8640457184f756a9c457f819 [term 1 LEADER]: Becoming Leader. State: Replica: 5288a8df8640457184f756a9c457f819, State: Running, Role: LEADER
I20260929 03:19:31.125170 2806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5288a8df8640457184f756a9c457f819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5288a8df8640457184f756a9c457f819" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32783 } }
I20260929 03:19:31.125895 2801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5288a8df8640457184f756a9c457f819 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:31.127041 2808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5288a8df8640457184f756a9c457f819 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5288a8df8640457184f756a9c457f819" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5288a8df8640457184f756a9c457f819" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32783 } } }
I20260929 03:19:31.127377 2808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5288a8df8640457184f756a9c457f819 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:31.127292 2807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5288a8df8640457184f756a9c457f819 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5288a8df8640457184f756a9c457f819. Latest consensus state: current_term: 1 leader_uuid: "5288a8df8640457184f756a9c457f819" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5288a8df8640457184f756a9c457f819" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32783 } } }
I20260929 03:19:31.127696 2807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5288a8df8640457184f756a9c457f819 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:31.128279 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:31.114938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:41677 (local address 127.0.0.1:32783)
0929 03:19:31.115388 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:31.115403 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:31.115439 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:19:31.116105 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:31.116126 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:31.116500 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:31.116826 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:31.117676 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:31.118727 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:31.119586 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:31.120075 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:31.120223 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:31.122836 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:31.122841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:31.122855 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:31.122884 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:31.124497 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:31.125088 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:31.125094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:31.125101 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:31.125240 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:31.125586 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:31.125589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:31.125591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:31.126036 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:31.126270 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:31.126903 (+ 633us) server_negotiation.cc:299] Negotiation successful
0929 03:19:31.127147 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":140,"threads_started":1}
I20260929 03:19:31.128564 2818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:31.131583 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:19:31.131680 2818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:31.139015 2818 catalog_manager.cc:1347] Generated new cluster ID: f59d98cf55d049059bd16b04053be50d
I20260929 03:19:31.139087 2818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:31.142735 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:31.133858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:51949 (local address 127.0.0.1:32783)
0929 03:19:31.133976 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:31.133981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:31.134241 (+ 260us) server_negotiation.cc:407] Connection header received
0929 03:19:31.134388 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:31.134391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:31.134441 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:31.134525 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:31.135321 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:31.136335 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:31.137288 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:31.137525 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:31.137611 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:31.139001 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:31.139005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:31.139008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:31.139035 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:31.140663 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:31.141195 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:31.141199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:31.141201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:31.141282 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:31.141582 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:31.141585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:31.141587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:31.141774 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:31.141903 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:31.142493 (+ 590us) server_negotiation.cc:299] Negotiation successful
0929 03:19:31.142582 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:19:31.151265 2818 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:31.152163 2818 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:31.161222 2818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5288a8df8640457184f756a9c457f819: Generated new TSK 0
I20260929 03:19:31.161901 2818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:19:31.168707 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 2732
W20260929 03:19:31.173804 2717 connection.cc:535] client connection to 127.0.0.1:32783 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32783): Connection reset by peer (error 104)
2026-09-29T03:19:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:19:31 dist-test-slave-nbjg krb5kdc[2833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:19:31 dist-test-slave-nbjg krb5kdc[2833](info): set up 2 sockets
Sep 29 03:19:31 dist-test-slave-nbjg krb5kdc[2833](info): commencing operation
krb5kdc: starting...
W20260929 03:19:32.122450 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:19:32 dist-test-slave-nbjg krb5kdc[2833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:19:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:19:32.238234 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41823
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:32.346526 2846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:32.346778 2846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:32.346817 2846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:32.350759 2846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:32.350814 2846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:32.350831 2846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:32.350847 2846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:32.350865 2846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:32.356046 2846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:32.357138 2846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:32.358147 2846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:32.366245 2852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:32.366307 2855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:32.366245 2851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:32.366418 2846 server_base.cc:1034] running on GCE node
I20260929 03:19:32.366984 2846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:32.368067 2846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:32.369248 2846 hybrid_clock.cc:648] HybridClock initialized: now 1790651972369229 us; error 29 us; skew 500 ppm
Sep 29 03:19:32 dist-test-slave-nbjg krb5kdc[2833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:32.378321 2846 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:19:32.379305 2846 webserver.cc:460] Webserver started at http://127.0.0.1:33661/ using document root <none> and password file <none>
I20260929 03:19:32.379871 2846 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:32.379938 2846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:32.380146 2846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:32.382216 2846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "7e96930e923f4a8598d35a6fc5da58e5"
format_stamp: "Formatted at 2026-09-29 03:19:32 on dist-test-slave-nbjg"
server_key: "41d3574bb29bcfe589e1d5f68899f57f"
server_key_iv: "2aa125be33cd9f195998380aad1a7010"
server_key_version: "encryptionkey@0"
I20260929 03:19:32.382771 2846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "7e96930e923f4a8598d35a6fc5da58e5"
format_stamp: "Formatted at 2026-09-29 03:19:32 on dist-test-slave-nbjg"
server_key: "41d3574bb29bcfe589e1d5f68899f57f"
server_key_iv: "2aa125be33cd9f195998380aad1a7010"
server_key_version: "encryptionkey@0"
I20260929 03:19:32.386868 2846 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:19:32.389678 2862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:32.391206 2846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:19:32.391353 2846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "7e96930e923f4a8598d35a6fc5da58e5"
format_stamp: "Formatted at 2026-09-29 03:19:32 on dist-test-slave-nbjg"
server_key: "41d3574bb29bcfe589e1d5f68899f57f"
server_key_iv: "2aa125be33cd9f195998380aad1a7010"
server_key_version: "encryptionkey@0"
I20260929 03:19:32.391470 2846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:32.410528 2846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:32.413352 2846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:32.413535 2846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:32.420624 2846 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41823
I20260929 03:19:32.420642 2914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41823 every 8 connection(s)
I20260929 03:19:32.421598 2846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:32.424158 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 2846
I20260929 03:19:32.424278 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:19:32.424494 2915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:32.424530 14266 external_mini_cluster.cc:1428] Setting key 6bf97d6198b1e5cfa3cbffdca2b3df55
I20260929 03:19:32.431793 2915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e96930e923f4a8598d35a6fc5da58e5: Bootstrap starting.
I20260929 03:19:32.434474 2915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e96930e923f4a8598d35a6fc5da58e5: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:32.434710 2918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:32.425907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:46663 (local address 127.0.0.1:41823)
0929 03:19:32.426368 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:32.426386 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:32.426430 (+ 44us) server_negotiation.cc:407] Connection header received
0929 03:19:32.427253 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:32.427279 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:32.427754 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:32.428143 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:32.428564 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:32.430043 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:32.431022 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:32.431709 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:32.431864 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:32.432678 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:32.432685 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:32.432701 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:32.433201 (+ 500us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:32.433269 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:32.433300 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:32.433419 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:32.433588 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 03:19:32.433730 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":119,"threads_started":1}
I20260929 03:19:32.435295 2915 log.cc:824] T 00000000000000000000000000000000 P 7e96930e923f4a8598d35a6fc5da58e5: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:32.437271 2915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e96930e923f4a8598d35a6fc5da58e5: No bootstrap required, opened a new log
I20260929 03:19:32.437587 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:19:32.441299 2915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e96930e923f4a8598d35a6fc5da58e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e96930e923f4a8598d35a6fc5da58e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41823 } }
I20260929 03:19:32.441576 2915 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e96930e923f4a8598d35a6fc5da58e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:32.441638 2915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e96930e923f4a8598d35a6fc5da58e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e96930e923f4a8598d35a6fc5da58e5, State: Initialized, Role: FOLLOWER
I20260929 03:19:32.442101 2915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e96930e923f4a8598d35a6fc5da58e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e96930e923f4a8598d35a6fc5da58e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41823 } }
I20260929 03:19:32.442296 2915 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e96930e923f4a8598d35a6fc5da58e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:32.442369 2915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e96930e923f4a8598d35a6fc5da58e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:32.442744 2915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e96930e923f4a8598d35a6fc5da58e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:32.443322 2918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:32.439486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:51595 (local address 127.0.0.1:41823)
0929 03:19:32.439593 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:32.439597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:32.439833 (+ 236us) server_negotiation.cc:407] Connection header received
0929 03:19:32.439939 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:32.439941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:32.439981 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:32.440038 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:32.440296 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:32.440772 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:32.441806 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:32.442020 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:32.442092 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:32.442921 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:32.442924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:32.442926 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:32.442937 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:32.442960 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:32.442970 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:32.443034 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:32.443171 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 03:19:32.443200 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 03:19:32.443812 2915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e96930e923f4a8598d35a6fc5da58e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e96930e923f4a8598d35a6fc5da58e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41823 } }
I20260929 03:19:32.444146 2915 leader_election.cc:302] T 00000000000000000000000000000000 P 7e96930e923f4a8598d35a6fc5da58e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e96930e923f4a8598d35a6fc5da58e5; no voters:
I20260929 03:19:32.444482 2915 leader_election.cc:288] T 00000000000000000000000000000000 P 7e96930e923f4a8598d35a6fc5da58e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:19:32.444554 2925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e96930e923f4a8598d35a6fc5da58e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:32.444759 2925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e96930e923f4a8598d35a6fc5da58e5 [term 1 LEADER]: Becoming Leader. State: Replica: 7e96930e923f4a8598d35a6fc5da58e5, State: Running, Role: LEADER
I20260929 03:19:32.445104 2925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e96930e923f4a8598d35a6fc5da58e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e96930e923f4a8598d35a6fc5da58e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41823 } }
I20260929 03:19:32.445726 2915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e96930e923f4a8598d35a6fc5da58e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:32.446806 2926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e96930e923f4a8598d35a6fc5da58e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e96930e923f4a8598d35a6fc5da58e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e96930e923f4a8598d35a6fc5da58e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41823 } } }
I20260929 03:19:32.446821 2927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e96930e923f4a8598d35a6fc5da58e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e96930e923f4a8598d35a6fc5da58e5. Latest consensus state: current_term: 1 leader_uuid: "7e96930e923f4a8598d35a6fc5da58e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e96930e923f4a8598d35a6fc5da58e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41823 } } }
I20260929 03:19:32.446946 2927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e96930e923f4a8598d35a6fc5da58e5 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:32.446947 2926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e96930e923f4a8598d35a6fc5da58e5 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:32.447374 2934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:32.450563 2934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:32.456275 2934 catalog_manager.cc:1347] Generated new cluster ID: 241919aa92994629a0b2b465e5028007
I20260929 03:19:32.456350 2934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:32.475315 2934 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:32.476248 2934 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:32.484082 2934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e96930e923f4a8598d35a6fc5da58e5: Generated new TSK 0
I20260929 03:19:32.484791 2934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:19:32.492247 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 2846
W20260929 03:19:32.497305 2831 connection.cc:535] client connection to 127.0.0.1:41823 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41823): Connection reset by peer (error 104)
2026-09-29T03:19:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:19:32 dist-test-slave-nbjg krb5kdc[2946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:19:32 dist-test-slave-nbjg krb5kdc[2946](info): set up 2 sockets
Sep 29 03:19:32 dist-test-slave-nbjg krb5kdc[2946](info): commencing operation
krb5kdc: starting...
W20260929 03:19:33.438398 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:19:33 dist-test-slave-nbjg krb5kdc[2946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:19:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:19:33.556994 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41521
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:33.667557 2959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:33.667821 2959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:33.667860 2959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:33.671797 2959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:33.671866 2959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:33.671885 2959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:33.671900 2959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:33.671912 2959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:33.677246 2959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:33.678342 2959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:33.679347 2959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:33.688236 2967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:33.688242 2965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:33.688236 2964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:33.688297 2959 server_base.cc:1034] running on GCE node
I20260929 03:19:33.689038 2959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:33.690325 2959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:33.691494 2959 hybrid_clock.cc:648] HybridClock initialized: now 1790651973691477 us; error 33 us; skew 500 ppm
Sep 29 03:19:33 dist-test-slave-nbjg krb5kdc[2946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:33.694116 2959 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:19:33.695020 2959 webserver.cc:460] Webserver started at http://127.0.0.1:40637/ using document root <none> and password file <none>
I20260929 03:19:33.695600 2959 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:33.695698 2959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:33.695909 2959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:33.698125 2959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "004d2fad8fa84b0a94ae62565a8c5cad"
format_stamp: "Formatted at 2026-09-29 03:19:33 on dist-test-slave-nbjg"
server_key: "c249bf73eb6099d13d980b531234ec6a"
server_key_iv: "cc4c5897d477d3d28a24b19da1bc7ff2"
server_key_version: "encryptionkey@0"
I20260929 03:19:33.698740 2959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "004d2fad8fa84b0a94ae62565a8c5cad"
format_stamp: "Formatted at 2026-09-29 03:19:33 on dist-test-slave-nbjg"
server_key: "c249bf73eb6099d13d980b531234ec6a"
server_key_iv: "cc4c5897d477d3d28a24b19da1bc7ff2"
server_key_version: "encryptionkey@0"
I20260929 03:19:33.702653 2959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:19:33.705121 2974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:33.706437 2959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:19:33.706552 2959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "004d2fad8fa84b0a94ae62565a8c5cad"
format_stamp: "Formatted at 2026-09-29 03:19:33 on dist-test-slave-nbjg"
server_key: "c249bf73eb6099d13d980b531234ec6a"
server_key_iv: "cc4c5897d477d3d28a24b19da1bc7ff2"
server_key_version: "encryptionkey@0"
I20260929 03:19:33.706661 2959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:33.711230 2959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:33.713990 2959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:33.714159 2959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:33.721287 2959 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41521
I20260929 03:19:33.721307 3026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41521 every 8 connection(s)
I20260929 03:19:33.722272 2959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:33.722756 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 2959
I20260929 03:19:33.722901 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:19:33.723210 14266 external_mini_cluster.cc:1428] Setting key e8639559c14ab3fb17b22179381ec640
I20260929 03:19:33.725534 3027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:33.728272 3030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:33.724631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:56503 (local address 127.0.0.1:41521)
0929 03:19:33.725061 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:33.725075 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:33.725112 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:19:33.725795 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:33.725814 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:33.726103 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:33.726452 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:33.727159 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:33.727164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:33.727179 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:33.727190 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:33.727240 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:33.727258 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:33.727379 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:33.727447 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 03:19:33.727598 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":137,"threads_started":1}
I20260929 03:19:33.731045 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:19:33.731798 3027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 004d2fad8fa84b0a94ae62565a8c5cad: Bootstrap starting.
I20260929 03:19:33.734278 3027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 004d2fad8fa84b0a94ae62565a8c5cad: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:33.734449 3030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:33.732934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:36435 (local address 127.0.0.1:41521)
0929 03:19:33.733030 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:33.733034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:33.733244 (+ 210us) server_negotiation.cc:407] Connection header received
0929 03:19:33.733346 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:33.733349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:33.733381 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:33.733438 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:33.734064 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:33.734067 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:33.734070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:33.734081 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:33.734107 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:33.734124 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:33.734234 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:33.734296 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 03:19:33.734335 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 03:19:33.735153 3027 log.cc:824] T 00000000000000000000000000000000 P 004d2fad8fa84b0a94ae62565a8c5cad: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:33.737267 3027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 004d2fad8fa84b0a94ae62565a8c5cad: No bootstrap required, opened a new log
I20260929 03:19:33.740788 3027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 004d2fad8fa84b0a94ae62565a8c5cad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004d2fad8fa84b0a94ae62565a8c5cad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41521 } }
I20260929 03:19:33.741017 3027 raft_consensus.cc:374] T 00000000000000000000000000000000 P 004d2fad8fa84b0a94ae62565a8c5cad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:33.741056 3027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 004d2fad8fa84b0a94ae62565a8c5cad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 004d2fad8fa84b0a94ae62565a8c5cad, State: Initialized, Role: FOLLOWER
I20260929 03:19:33.741535 3027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 004d2fad8fa84b0a94ae62565a8c5cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004d2fad8fa84b0a94ae62565a8c5cad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41521 } }
I20260929 03:19:33.741711 3027 raft_consensus.cc:388] T 00000000000000000000000000000000 P 004d2fad8fa84b0a94ae62565a8c5cad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:33.741814 3027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 004d2fad8fa84b0a94ae62565a8c5cad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:33.741922 3027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 004d2fad8fa84b0a94ae62565a8c5cad [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:33.742959 3027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 004d2fad8fa84b0a94ae62565a8c5cad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004d2fad8fa84b0a94ae62565a8c5cad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41521 } }
I20260929 03:19:33.743283 3027 leader_election.cc:302] T 00000000000000000000000000000000 P 004d2fad8fa84b0a94ae62565a8c5cad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 004d2fad8fa84b0a94ae62565a8c5cad; no voters:
I20260929 03:19:33.743554 3027 leader_election.cc:288] T 00000000000000000000000000000000 P 004d2fad8fa84b0a94ae62565a8c5cad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:19:33.743714 3037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 004d2fad8fa84b0a94ae62565a8c5cad [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:33.743909 3037 raft_consensus.cc:686] T 00000000000000000000000000000000 P 004d2fad8fa84b0a94ae62565a8c5cad [term 1 LEADER]: Becoming Leader. State: Replica: 004d2fad8fa84b0a94ae62565a8c5cad, State: Running, Role: LEADER
I20260929 03:19:33.744258 3037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 004d2fad8fa84b0a94ae62565a8c5cad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004d2fad8fa84b0a94ae62565a8c5cad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41521 } }
I20260929 03:19:33.744787 3027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 004d2fad8fa84b0a94ae62565a8c5cad [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:33.745836 3038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 004d2fad8fa84b0a94ae62565a8c5cad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "004d2fad8fa84b0a94ae62565a8c5cad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004d2fad8fa84b0a94ae62565a8c5cad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41521 } } }
I20260929 03:19:33.745996 3038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 004d2fad8fa84b0a94ae62565a8c5cad [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:33.746574 3039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 004d2fad8fa84b0a94ae62565a8c5cad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 004d2fad8fa84b0a94ae62565a8c5cad. Latest consensus state: current_term: 1 leader_uuid: "004d2fad8fa84b0a94ae62565a8c5cad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004d2fad8fa84b0a94ae62565a8c5cad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41521 } } }
I20260929 03:19:33.746668 3039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 004d2fad8fa84b0a94ae62565a8c5cad [sys.catalog]: This master's current role is: LEADER
W20260929 03:19:33.749457 3052 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 004d2fad8fa84b0a94ae62565a8c5cad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:19:33.749521 3052 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:19:33.749598 3046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:33.750391 3046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:33.755827 3046 catalog_manager.cc:1347] Generated new cluster ID: e2309be07dc84ffd9989ad75af7f8138
I20260929 03:19:33.755956 3046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:33.769789 3046 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:33.770118 3046 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:33.776271 3046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 004d2fad8fa84b0a94ae62565a8c5cad: Generated new TSK 0
I20260929 03:19:33.776968 3046 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:19:33.787272 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 2959
2026-09-29T03:19:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:19:33 dist-test-slave-nbjg krb5kdc[3058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:19:33 dist-test-slave-nbjg krb5kdc[3058](info): set up 2 sockets
Sep 29 03:19:33 dist-test-slave-nbjg krb5kdc[3058](info): commencing operation
krb5kdc: starting...
W20260929 03:19:34.742802 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:19:34 dist-test-slave-nbjg krb5kdc[3058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:19:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:19:34.861017 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45339
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:34.969852 3071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:34.970161 3071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:34.970243 3071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:34.974327 3071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:34.974401 3071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:34.974427 3071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:34.974457 3071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:34.974478 3071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:34.979980 3071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45339
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:34.981094 3071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:34.982131 3071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:34.990394 3077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:34.990394 3076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:34.990404 3079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:34.990638 3071 server_base.cc:1034] running on GCE node
I20260929 03:19:34.991279 3071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:34.992340 3071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:34.993515 3071 hybrid_clock.cc:648] HybridClock initialized: now 1790651974993492 us; error 34 us; skew 500 ppm
Sep 29 03:19:34 dist-test-slave-nbjg krb5kdc[3058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:34.996202 3071 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:19:34.997104 3071 webserver.cc:460] Webserver started at http://127.0.0.1:33301/ using document root <none> and password file <none>
I20260929 03:19:34.997644 3071 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:34.997713 3071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:34.997918 3071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:35.000005 3071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "5aaf78deaad74dcd9819ddc487b58010"
format_stamp: "Formatted at 2026-09-29 03:19:34 on dist-test-slave-nbjg"
server_key: "3909f46c296cb2bff3c41e64441e408d"
server_key_iv: "594181ebd411b962d8ed4d729c4579d2"
server_key_version: "encryptionkey@0"
I20260929 03:19:35.000562 3071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "5aaf78deaad74dcd9819ddc487b58010"
format_stamp: "Formatted at 2026-09-29 03:19:34 on dist-test-slave-nbjg"
server_key: "3909f46c296cb2bff3c41e64441e408d"
server_key_iv: "594181ebd411b962d8ed4d729c4579d2"
server_key_version: "encryptionkey@0"
I20260929 03:19:35.004227 3071 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 03:19:35.006642 3086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:35.007872 3071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:19:35.007998 3071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "5aaf78deaad74dcd9819ddc487b58010"
format_stamp: "Formatted at 2026-09-29 03:19:34 on dist-test-slave-nbjg"
server_key: "3909f46c296cb2bff3c41e64441e408d"
server_key_iv: "594181ebd411b962d8ed4d729c4579d2"
server_key_version: "encryptionkey@0"
I20260929 03:19:35.008116 3071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:35.035928 3071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:35.038913 3071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:35.039093 3071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:35.045796 3071 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45339
I20260929 03:19:35.045811 3138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45339 every 8 connection(s)
I20260929 03:19:35.046768 3071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:35.047048 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 3071
I20260929 03:19:35.047190 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:19:35.047494 14266 external_mini_cluster.cc:1428] Setting key 1323de4603469895d9ee344e6e346aa7
I20260929 03:19:35.049847 3139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:35.057181 3139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aaf78deaad74dcd9819ddc487b58010: Bootstrap starting.
Sep 29 03:19:35 dist-test-slave-nbjg krb5kdc[3058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:19:35.060215 3139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5aaf78deaad74dcd9819ddc487b58010: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:35.061126 3139 log.cc:824] T 00000000000000000000000000000000 P 5aaf78deaad74dcd9819ddc487b58010: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:35.063421 3139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aaf78deaad74dcd9819ddc487b58010: No bootstrap required, opened a new log
I20260929 03:19:35.063812 3142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:35.048761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:47571 (local address 127.0.0.1:45339)
0929 03:19:35.049157 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:35.049173 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:35.049209 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:19:35.049848 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:35.049863 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:35.050281 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:35.050581 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:35.051453 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:35.052491 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:35.053724 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:35.054434 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:35.054603 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:35.058343 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:35.058350 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:35.058371 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:35.058414 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:35.060474 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:35.061191 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:35.061197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:35.061204 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:35.061349 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:35.061700 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:35.061704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:35.061706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:35.062157 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:35.062398 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:35.062746 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 03:19:35.062988 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":108,"threads_started":1}
I20260929 03:19:35.067253 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:19:35.067689 3139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aaf78deaad74dcd9819ddc487b58010 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaf78deaad74dcd9819ddc487b58010" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45339 } }
I20260929 03:19:35.067929 3139 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5aaf78deaad74dcd9819ddc487b58010 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:35.067979 3139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aaf78deaad74dcd9819ddc487b58010 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aaf78deaad74dcd9819ddc487b58010, State: Initialized, Role: FOLLOWER
I20260929 03:19:35.068478 3139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aaf78deaad74dcd9819ddc487b58010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaf78deaad74dcd9819ddc487b58010" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45339 } }
I20260929 03:19:35.068650 3139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5aaf78deaad74dcd9819ddc487b58010 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:35.068719 3139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5aaf78deaad74dcd9819ddc487b58010 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:35.068820 3139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aaf78deaad74dcd9819ddc487b58010 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:35.069936 3139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5aaf78deaad74dcd9819ddc487b58010 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaf78deaad74dcd9819ddc487b58010" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45339 } }
I20260929 03:19:35.070339 3139 leader_election.cc:302] T 00000000000000000000000000000000 P 5aaf78deaad74dcd9819ddc487b58010 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5aaf78deaad74dcd9819ddc487b58010; no voters:
I20260929 03:19:35.070652 3139 leader_election.cc:288] T 00000000000000000000000000000000 P 5aaf78deaad74dcd9819ddc487b58010 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:19:35.070765 3149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5aaf78deaad74dcd9819ddc487b58010 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:35.070932 3149 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5aaf78deaad74dcd9819ddc487b58010 [term 1 LEADER]: Becoming Leader. State: Replica: 5aaf78deaad74dcd9819ddc487b58010, State: Running, Role: LEADER
I20260929 03:19:35.071316 3149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aaf78deaad74dcd9819ddc487b58010 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaf78deaad74dcd9819ddc487b58010" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45339 } }
I20260929 03:19:35.072062 3139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aaf78deaad74dcd9819ddc487b58010 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:35.073072 3151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aaf78deaad74dcd9819ddc487b58010 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aaf78deaad74dcd9819ddc487b58010. Latest consensus state: current_term: 1 leader_uuid: "5aaf78deaad74dcd9819ddc487b58010" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaf78deaad74dcd9819ddc487b58010" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45339 } } }
I20260929 03:19:35.073210 3151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aaf78deaad74dcd9819ddc487b58010 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:35.073596 3158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:35.073536 3150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aaf78deaad74dcd9819ddc487b58010 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5aaf78deaad74dcd9819ddc487b58010" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aaf78deaad74dcd9819ddc487b58010" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45339 } } }
I20260929 03:19:35.073714 3150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aaf78deaad74dcd9819ddc487b58010 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:35.076967 3158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:35.078320 3142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:35.069353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:60251 (local address 127.0.0.1:45339)
0929 03:19:35.069472 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:35.069477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:35.069649 (+ 172us) server_negotiation.cc:407] Connection header received
0929 03:19:35.069782 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:35.069785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:35.069833 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:35.069892 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:35.070659 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:35.071395 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:35.072251 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:35.072524 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:35.072658 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:35.074399 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:35.074403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:35.074405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:35.074431 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:35.076017 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:35.076615 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:35.076618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:35.076620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:35.076709 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:35.077373 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:35.077376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:35.077378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:35.077600 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:35.077744 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:35.078038 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 03:19:35.078143 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:19:35.083153 3158 catalog_manager.cc:1347] Generated new cluster ID: ed1e465d49cf4ada983d76389fe68a0f
I20260929 03:19:35.083220 3158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:35.094769 3158 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:35.096621 3158 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:35.105832 3158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5aaf78deaad74dcd9819ddc487b58010: Generated new TSK 0
I20260929 03:19:35.106722 3158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:19:35.113132 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 3071
W20260929 03:19:35.118444 3056 connection.cc:535] client connection to 127.0.0.1:45339 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45339): Connection reset by peer (error 104)
2026-09-29T03:19:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:19:35 dist-test-slave-nbjg krb5kdc[3170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:19:35 dist-test-slave-nbjg krb5kdc[3170](info): set up 2 sockets
Sep 29 03:19:35 dist-test-slave-nbjg krb5kdc[3170](info): commencing operation
krb5kdc: starting...
W20260929 03:19:36.074410 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:19:36 dist-test-slave-nbjg krb5kdc[3170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:19:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:19:36.193868 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:36.301944 3183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:36.302241 3183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:36.302299 3183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:36.306244 3183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:36.306298 3183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:36.306314 3183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:36.306330 3183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:36.306344 3183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:36.311465 3183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46371
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:36.312443 3183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:36.313411 3183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:36.321851 3189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:36.321851 3188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:36.321992 3183 server_base.cc:1034] running on GCE node
W20260929 03:19:36.321868 3191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:36.322656 3183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:36.323781 3183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:36.324970 3183 hybrid_clock.cc:648] HybridClock initialized: now 1790651976324954 us; error 24 us; skew 500 ppm
Sep 29 03:19:36 dist-test-slave-nbjg krb5kdc[3170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:36.327687 3183 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:19:36.328533 3183 webserver.cc:460] Webserver started at http://127.0.0.1:44569/ using document root <none> and password file <none>
I20260929 03:19:36.329106 3183 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:36.329162 3183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:36.329339 3183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:36.331434 3183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "6d3bfbd8c38d4bcb846b449153be2da9"
format_stamp: "Formatted at 2026-09-29 03:19:36 on dist-test-slave-nbjg"
server_key: "029249bb69b8ae6805754aff65295fa7"
server_key_iv: "87c3a025c4413e50424669cc12722717"
server_key_version: "encryptionkey@0"
I20260929 03:19:36.331986 3183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "6d3bfbd8c38d4bcb846b449153be2da9"
format_stamp: "Formatted at 2026-09-29 03:19:36 on dist-test-slave-nbjg"
server_key: "029249bb69b8ae6805754aff65295fa7"
server_key_iv: "87c3a025c4413e50424669cc12722717"
server_key_version: "encryptionkey@0"
I20260929 03:19:36.335685 3183 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 03:19:36.338125 3198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:36.339402 3183 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:19:36.339516 3183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "6d3bfbd8c38d4bcb846b449153be2da9"
format_stamp: "Formatted at 2026-09-29 03:19:36 on dist-test-slave-nbjg"
server_key: "029249bb69b8ae6805754aff65295fa7"
server_key_iv: "87c3a025c4413e50424669cc12722717"
server_key_version: "encryptionkey@0"
I20260929 03:19:36.339617 3183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:36.366904 3183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:36.369827 3183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:36.370010 3183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:36.376760 3183 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46371
I20260929 03:19:36.376823 3250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46371 every 8 connection(s)
I20260929 03:19:36.377758 3183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:36.379783 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 3183
I20260929 03:19:36.379914 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:19:36.380178 14266 external_mini_cluster.cc:1428] Setting key 28b86391439284422f5f60d54f03758d
I20260929 03:19:36.380743 3251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:36.387771 3251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d3bfbd8c38d4bcb846b449153be2da9: Bootstrap starting.
I20260929 03:19:36.390512 3251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d3bfbd8c38d4bcb846b449153be2da9: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:36.390599 3254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:36.381588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:58907 (local address 127.0.0.1:46371)
0929 03:19:36.382029 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:36.382049 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:36.382091 (+ 42us) server_negotiation.cc:407] Connection header received
0929 03:19:36.382896 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:36.382921 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:36.383384 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:36.383767 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:36.384208 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:36.385671 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:36.386729 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:36.387412 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:36.387571 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:36.388430 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:36.388436 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:36.388452 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:36.388966 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:36.389040 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:36.389074 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:36.389208 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:36.389451 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 03:19:36.389591 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":118,"threads_started":1}
I20260929 03:19:36.391307 3251 log.cc:824] T 00000000000000000000000000000000 P 6d3bfbd8c38d4bcb846b449153be2da9: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:36.393352 3251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d3bfbd8c38d4bcb846b449153be2da9: No bootstrap required, opened a new log
I20260929 03:19:36.393605 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:19:36.397126 3251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d3bfbd8c38d4bcb846b449153be2da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d3bfbd8c38d4bcb846b449153be2da9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46371 } }
I20260929 03:19:36.397374 3251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d3bfbd8c38d4bcb846b449153be2da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:36.397442 3251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d3bfbd8c38d4bcb846b449153be2da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d3bfbd8c38d4bcb846b449153be2da9, State: Initialized, Role: FOLLOWER
I20260929 03:19:36.397863 3251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d3bfbd8c38d4bcb846b449153be2da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d3bfbd8c38d4bcb846b449153be2da9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46371 } }
I20260929 03:19:36.397992 3251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d3bfbd8c38d4bcb846b449153be2da9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:36.398057 3251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d3bfbd8c38d4bcb846b449153be2da9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:36.398156 3251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d3bfbd8c38d4bcb846b449153be2da9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:36.399096 3251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d3bfbd8c38d4bcb846b449153be2da9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d3bfbd8c38d4bcb846b449153be2da9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46371 } }
I20260929 03:19:36.399412 3254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:36.395597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:47391 (local address 127.0.0.1:46371)
0929 03:19:36.395716 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:36.395721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:36.395914 (+ 193us) server_negotiation.cc:407] Connection header received
0929 03:19:36.396021 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:36.396024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:36.396066 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:36.396137 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:36.396391 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:36.396885 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:36.397768 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:36.397981 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:36.398066 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:36.398982 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:36.398986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:36.398988 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:36.399000 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:36.399027 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:36.399037 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:36.399106 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:36.399240 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 03:19:36.399277 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:19:36.399420 3251 leader_election.cc:302] T 00000000000000000000000000000000 P 6d3bfbd8c38d4bcb846b449153be2da9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d3bfbd8c38d4bcb846b449153be2da9; no voters:
I20260929 03:19:36.399811 3251 leader_election.cc:288] T 00000000000000000000000000000000 P 6d3bfbd8c38d4bcb846b449153be2da9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:19:36.399902 3261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d3bfbd8c38d4bcb846b449153be2da9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:36.400064 3261 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d3bfbd8c38d4bcb846b449153be2da9 [term 1 LEADER]: Becoming Leader. State: Replica: 6d3bfbd8c38d4bcb846b449153be2da9, State: Running, Role: LEADER
I20260929 03:19:36.400424 3261 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d3bfbd8c38d4bcb846b449153be2da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d3bfbd8c38d4bcb846b449153be2da9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46371 } }
I20260929 03:19:36.401180 3251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d3bfbd8c38d4bcb846b449153be2da9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:36.401681 3261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d3bfbd8c38d4bcb846b449153be2da9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d3bfbd8c38d4bcb846b449153be2da9. Latest consensus state: current_term: 1 leader_uuid: "6d3bfbd8c38d4bcb846b449153be2da9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d3bfbd8c38d4bcb846b449153be2da9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46371 } } }
I20260929 03:19:36.401847 3261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d3bfbd8c38d4bcb846b449153be2da9 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:36.402060 3262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d3bfbd8c38d4bcb846b449153be2da9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d3bfbd8c38d4bcb846b449153be2da9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d3bfbd8c38d4bcb846b449153be2da9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46371 } } }
I20260929 03:19:36.402155 3262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d3bfbd8c38d4bcb846b449153be2da9 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:36.402352 3268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:36.405670 3268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:36.411496 3268 catalog_manager.cc:1347] Generated new cluster ID: 8a570909d3b944889ba10db3fc0c7bc6
I20260929 03:19:36.411559 3268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:36.424924 3268 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:36.425887 3268 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:36.429409 3268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d3bfbd8c38d4bcb846b449153be2da9: Generated new TSK 0
I20260929 03:19:36.430269 3268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:19:36.443050 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 3183
W20260929 03:19:36.447992 3168 connection.cc:535] client connection to 127.0.0.1:46371 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46371): Connection reset by peer (error 104)
2026-09-29T03:19:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:19:36 dist-test-slave-nbjg krb5kdc[3282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:19:36 dist-test-slave-nbjg krb5kdc[3282](info): set up 2 sockets
Sep 29 03:19:36 dist-test-slave-nbjg krb5kdc[3282](info): commencing operation
krb5kdc: starting...
W20260929 03:19:37.394500 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:19:37 dist-test-slave-nbjg krb5kdc[3282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:19:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:19:37.513056 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44693
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:37.624037 3295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:37.624333 3295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:37.624377 3295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:37.628549 3295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:37.628625 3295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:37.628644 3295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:37.628660 3295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:37.628674 3295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:37.634305 3295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44693
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:37.635455 3295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:37.636538 3295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:37.645268 3301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:37.645228 3303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:37.645217 3300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:37.645637 3295 server_base.cc:1034] running on GCE node
I20260929 03:19:37.646291 3295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:37.647439 3295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:37.648626 3295 hybrid_clock.cc:648] HybridClock initialized: now 1790651977648608 us; error 31 us; skew 500 ppm
Sep 29 03:19:37 dist-test-slave-nbjg krb5kdc[3282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:37.655730 3295 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:19:37.656713 3295 webserver.cc:460] Webserver started at http://127.0.0.1:42709/ using document root <none> and password file <none>
I20260929 03:19:37.657269 3295 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:37.657330 3295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:37.657536 3295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:37.659652 3295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "687806f1f8404b6d9e143a2223d76b23"
format_stamp: "Formatted at 2026-09-29 03:19:37 on dist-test-slave-nbjg"
server_key: "85f7dadb42faa5c6bc4a4b4d52aaaac4"
server_key_iv: "4f0af0ec86a31498b1e2d7b28745eb0b"
server_key_version: "encryptionkey@0"
I20260929 03:19:37.660172 3295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "687806f1f8404b6d9e143a2223d76b23"
format_stamp: "Formatted at 2026-09-29 03:19:37 on dist-test-slave-nbjg"
server_key: "85f7dadb42faa5c6bc4a4b4d52aaaac4"
server_key_iv: "4f0af0ec86a31498b1e2d7b28745eb0b"
server_key_version: "encryptionkey@0"
I20260929 03:19:37.663975 3295 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 03:19:37.666425 3310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:37.667799 3295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:19:37.667954 3295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "687806f1f8404b6d9e143a2223d76b23"
format_stamp: "Formatted at 2026-09-29 03:19:37 on dist-test-slave-nbjg"
server_key: "85f7dadb42faa5c6bc4a4b4d52aaaac4"
server_key_iv: "4f0af0ec86a31498b1e2d7b28745eb0b"
server_key_version: "encryptionkey@0"
I20260929 03:19:37.668072 3295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:37.672729 3295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:37.675678 3295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:37.675832 3295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:37.682929 3295 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44693
I20260929 03:19:37.683079 3362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44693 every 8 connection(s)
I20260929 03:19:37.683930 3295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:37.687099 3363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:37.689209 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 3295
I20260929 03:19:37.689321 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:19:37.689604 14266 external_mini_cluster.cc:1428] Setting key afddf0f168d08fec96606167788080ee
I20260929 03:19:37.693761 3363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 687806f1f8404b6d9e143a2223d76b23: Bootstrap starting.
I20260929 03:19:37.695613 3366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:37.691062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:50267 (local address 127.0.0.1:44693)
0929 03:19:37.691516 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:37.691536 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:37.691583 (+ 47us) server_negotiation.cc:407] Connection header received
0929 03:19:37.692551 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:37.692578 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:37.692982 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:37.693373 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:37.694086 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:37.694092 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:37.694108 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:37.694120 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:37.694221 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:37.694253 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:37.694393 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:37.694492 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 03:19:37.694660 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":124,"threads_started":1}
I20260929 03:19:37.696643 3363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 687806f1f8404b6d9e143a2223d76b23: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:37.697453 3363 log.cc:824] T 00000000000000000000000000000000 P 687806f1f8404b6d9e143a2223d76b23: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:37.698689 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:19:37.699643 3363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 687806f1f8404b6d9e143a2223d76b23: No bootstrap required, opened a new log
I20260929 03:19:37.702329 3366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:37.700696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:34005 (local address 127.0.0.1:44693)
0929 03:19:37.700799 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:37.700803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:37.701033 (+ 230us) server_negotiation.cc:407] Connection header received
0929 03:19:37.701148 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:37.701150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:37.701186 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:37.701252 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:37.701932 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:37.701936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:37.701938 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:37.701950 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:37.701986 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:37.701997 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:37.702067 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:37.702152 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 03:19:37.702190 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:19:37.703586 3363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 687806f1f8404b6d9e143a2223d76b23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687806f1f8404b6d9e143a2223d76b23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44693 } }
I20260929 03:19:37.703830 3363 raft_consensus.cc:374] T 00000000000000000000000000000000 P 687806f1f8404b6d9e143a2223d76b23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:37.703879 3363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 687806f1f8404b6d9e143a2223d76b23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 687806f1f8404b6d9e143a2223d76b23, State: Initialized, Role: FOLLOWER
I20260929 03:19:37.704335 3363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 687806f1f8404b6d9e143a2223d76b23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687806f1f8404b6d9e143a2223d76b23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44693 } }
I20260929 03:19:37.704473 3363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 687806f1f8404b6d9e143a2223d76b23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:37.704530 3363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 687806f1f8404b6d9e143a2223d76b23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:37.704628 3363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 687806f1f8404b6d9e143a2223d76b23 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:37.705574 3363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 687806f1f8404b6d9e143a2223d76b23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687806f1f8404b6d9e143a2223d76b23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44693 } }
I20260929 03:19:37.705920 3363 leader_election.cc:302] T 00000000000000000000000000000000 P 687806f1f8404b6d9e143a2223d76b23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 687806f1f8404b6d9e143a2223d76b23; no voters:
I20260929 03:19:37.706270 3363 leader_election.cc:288] T 00000000000000000000000000000000 P 687806f1f8404b6d9e143a2223d76b23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:19:37.706398 3373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 687806f1f8404b6d9e143a2223d76b23 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:37.706607 3373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 687806f1f8404b6d9e143a2223d76b23 [term 1 LEADER]: Becoming Leader. State: Replica: 687806f1f8404b6d9e143a2223d76b23, State: Running, Role: LEADER
I20260929 03:19:37.706954 3373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 687806f1f8404b6d9e143a2223d76b23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687806f1f8404b6d9e143a2223d76b23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44693 } }
I20260929 03:19:37.707509 3363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 687806f1f8404b6d9e143a2223d76b23 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:37.708573 3375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 687806f1f8404b6d9e143a2223d76b23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 687806f1f8404b6d9e143a2223d76b23. Latest consensus state: current_term: 1 leader_uuid: "687806f1f8404b6d9e143a2223d76b23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687806f1f8404b6d9e143a2223d76b23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44693 } } }
I20260929 03:19:37.708921 3375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 687806f1f8404b6d9e143a2223d76b23 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:37.708815 3374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 687806f1f8404b6d9e143a2223d76b23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "687806f1f8404b6d9e143a2223d76b23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687806f1f8404b6d9e143a2223d76b23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44693 } } }
I20260929 03:19:37.709175 3374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 687806f1f8404b6d9e143a2223d76b23 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:37.709348 3383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:37.712306 3383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:37.717950 3383 catalog_manager.cc:1347] Generated new cluster ID: 49e6765b706f4c1b93678d7d446b80d2
I20260929 03:19:37.718030 3383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:37.729897 3383 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:37.730347 3383 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:37.735877 3383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 687806f1f8404b6d9e143a2223d76b23: Generated new TSK 0
I20260929 03:19:37.736665 3383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:19:37.742375 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 3295
2026-09-29T03:19:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:19:37 dist-test-slave-nbjg krb5kdc[3395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:19:37 dist-test-slave-nbjg krb5kdc[3395](info): set up 2 sockets
Sep 29 03:19:37 dist-test-slave-nbjg krb5kdc[3395](info): commencing operation
krb5kdc: starting...
W20260929 03:19:38.686462 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:19:38 dist-test-slave-nbjg krb5kdc[3395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:19:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:19:38.803323 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:42871
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:45631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:42871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:38.911617 3408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:38.911870 3408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:38.911911 3408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:38.915867 3408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:38.915927 3408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:38.915946 3408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:38.915961 3408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:38.915973 3408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:38.921129 3408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:45631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:42871
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:42871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:38.922134 3408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:38.923128 3408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:19:38.931413 3408 server_base.cc:1034] running on GCE node
W20260929 03:19:38.931423 3414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:38.931424 3413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:38.931617 3416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:38.932214 3408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:38.933308 3408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:38.934470 3408 hybrid_clock.cc:648] HybridClock initialized: now 1790651978934450 us; error 30 us; skew 500 ppm
Sep 29 03:19:38 dist-test-slave-nbjg krb5kdc[3395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651978, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:38.937220 3408 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:19:38.938126 3408 webserver.cc:460] Webserver started at http://127.13.238.190:37529/ using document root <none> and password file <none>
I20260929 03:19:38.938686 3408 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:38.938746 3408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:38.938941 3408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:38.940970 3408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "932c59421b0949f2a7de17497a3dca58"
format_stamp: "Formatted at 2026-09-29 03:19:38 on dist-test-slave-nbjg"
server_key: "88736810b6327d3831d8f773457c1781"
server_key_iv: "be1de8419e5ab66c9640b26673a27656"
server_key_version: "encryptionkey@0"
I20260929 03:19:38.941505 3408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "932c59421b0949f2a7de17497a3dca58"
format_stamp: "Formatted at 2026-09-29 03:19:38 on dist-test-slave-nbjg"
server_key: "88736810b6327d3831d8f773457c1781"
server_key_iv: "be1de8419e5ab66c9640b26673a27656"
server_key_version: "encryptionkey@0"
I20260929 03:19:38.945168 3408 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 03:19:38.947603 3423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:38.948910 3408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 03:19:38.949024 3408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "932c59421b0949f2a7de17497a3dca58"
format_stamp: "Formatted at 2026-09-29 03:19:38 on dist-test-slave-nbjg"
server_key: "88736810b6327d3831d8f773457c1781"
server_key_iv: "be1de8419e5ab66c9640b26673a27656"
server_key_version: "encryptionkey@0"
I20260929 03:19:38.949124 3408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:38.965147 3408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:38.968243 3408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:38.968423 3408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:38.975347 3408 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:42871
I20260929 03:19:38.975363 3476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:42871 every 8 connection(s)
I20260929 03:19:38.976303 3408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:38.978780 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 3408
I20260929 03:19:38.978914 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:19:38.979210 14266 external_mini_cluster.cc:1428] Setting key a259423a9c1857121bf2dd596f563dab
I20260929 03:19:38.979401 3477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:38.986559 3477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 932c59421b0949f2a7de17497a3dca58: Bootstrap starting.
Sep 29 03:19:38 dist-test-slave-nbjg krb5kdc[3395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:19:38.989248 3477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 932c59421b0949f2a7de17497a3dca58: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:38.989930 3477 log.cc:824] T 00000000000000000000000000000000 P 932c59421b0949f2a7de17497a3dca58: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:38.991856 3477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 932c59421b0949f2a7de17497a3dca58: No bootstrap required, opened a new log
I20260929 03:19:38.993538 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:38.980567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42677 (local address 127.13.238.190:42871)
0929 03:19:38.980949 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:38.980964 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:38.981003 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:19:38.981639 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:38.981656 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:38.981987 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:38.982297 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:38.983165 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:38.984154 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:38.985355 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:38.985844 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:38.985959 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:38.988809 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:38.988815 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:38.988829 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:38.988857 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:38.990541 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:38.991044 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:38.991049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:38.991054 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:38.991166 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:38.991469 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:38.991472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:38.991473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:38.991874 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:38.992073 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:38.992389 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 03:19:38.992614 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":103,"threads_started":1}
I20260929 03:19:38.995169 3477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 932c59421b0949f2a7de17497a3dca58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932c59421b0949f2a7de17497a3dca58" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42871 } }
I20260929 03:19:38.995434 3477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 932c59421b0949f2a7de17497a3dca58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:38.995509 3477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 932c59421b0949f2a7de17497a3dca58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 932c59421b0949f2a7de17497a3dca58, State: Initialized, Role: FOLLOWER
I20260929 03:19:38.995970 3477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 932c59421b0949f2a7de17497a3dca58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932c59421b0949f2a7de17497a3dca58" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42871 } }
I20260929 03:19:38.996107 3477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 932c59421b0949f2a7de17497a3dca58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:38.996172 3477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 932c59421b0949f2a7de17497a3dca58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:38.996265 3477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 932c59421b0949f2a7de17497a3dca58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:38.996610 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:19:38.997398 3477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 932c59421b0949f2a7de17497a3dca58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932c59421b0949f2a7de17497a3dca58" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42871 } }
I20260929 03:19:38.997776 3477 leader_election.cc:302] T 00000000000000000000000000000000 P 932c59421b0949f2a7de17497a3dca58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 932c59421b0949f2a7de17497a3dca58; no voters:
I20260929 03:19:38.998124 3477 leader_election.cc:288] T 00000000000000000000000000000000 P 932c59421b0949f2a7de17497a3dca58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:19:38.998212 3486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 932c59421b0949f2a7de17497a3dca58 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:38.998417 3486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 932c59421b0949f2a7de17497a3dca58 [term 1 LEADER]: Becoming Leader. State: Replica: 932c59421b0949f2a7de17497a3dca58, State: Running, Role: LEADER
I20260929 03:19:38.999171 3486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 932c59421b0949f2a7de17497a3dca58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932c59421b0949f2a7de17497a3dca58" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42871 } }
I20260929 03:19:38.999460 3477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 932c59421b0949f2a7de17497a3dca58 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:39.000248 3486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932c59421b0949f2a7de17497a3dca58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 932c59421b0949f2a7de17497a3dca58. Latest consensus state: current_term: 1 leader_uuid: "932c59421b0949f2a7de17497a3dca58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932c59421b0949f2a7de17497a3dca58" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42871 } } }
I20260929 03:19:39.000442 3486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932c59421b0949f2a7de17497a3dca58 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:39.000630 3487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932c59421b0949f2a7de17497a3dca58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "932c59421b0949f2a7de17497a3dca58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932c59421b0949f2a7de17497a3dca58" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42871 } } }
I20260929 03:19:39.000713 3487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932c59421b0949f2a7de17497a3dca58 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:39.000869 3496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:39.003923 3496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:39.007649 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:38.998848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55047 (local address 127.13.238.190:42871)
0929 03:19:38.998961 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:38.998965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:38.999217 (+ 252us) server_negotiation.cc:407] Connection header received
0929 03:19:38.999356 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:38.999360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:38.999405 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:38.999463 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:39.000378 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:39.001154 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:39.002301 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:39.002628 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:39.002727 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:39.004232 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:39.004236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:39.004239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:39.004263 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:39.005778 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:39.006367 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:39.006371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:39.006373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:39.006462 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:39.006801 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:39.006804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:39.006806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:39.007008 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:39.007160 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:39.007408 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 03:19:39.007500 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:19:39.010064 3496 catalog_manager.cc:1347] Generated new cluster ID: 3067c863db044f09a026f3fd5c8ef989
I20260929 03:19:39.010133 3496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:39.025552 3496 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:39.026630 3496 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:39.034441 3496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 932c59421b0949f2a7de17497a3dca58: Generated new TSK 0
I20260929 03:19:39.035284 3496 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:19:39.049009 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 3408
W20260929 03:19:39.053881 3393 connection.cc:535] client connection to 127.13.238.190:42871 recv error: Network error: failed to read from TLS socket (remote: 127.13.238.190:42871): Connection reset by peer (error 104)
2026-09-29T03:19:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:19:39 dist-test-slave-nbjg krb5kdc[3509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:19:39 dist-test-slave-nbjg krb5kdc[3509](info): set up 2 sockets
Sep 29 03:19:39 dist-test-slave-nbjg krb5kdc[3509](info): commencing operation
krb5kdc: starting...
W20260929 03:19:39.998471 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:19:40 dist-test-slave-nbjg krb5kdc[3509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:19:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:19:40.116914 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:36815
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:36815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:40.224828 3522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:40.225100 3522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:40.225157 3522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:40.229166 3522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:40.229224 3522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:40.229251 3522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:40.229274 3522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:40.229301 3522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:40.234504 3522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:36815
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:36815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:40.235553 3522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:40.236572 3522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:40.244884 3528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:40.244869 3527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:40.244931 3522 server_base.cc:1034] running on GCE node
W20260929 03:19:40.244870 3530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:40.245690 3522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:40.246822 3522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:40.247972 3522 hybrid_clock.cc:648] HybridClock initialized: now 1790651980247945 us; error 40 us; skew 500 ppm
Sep 29 03:19:40 dist-test-slave-nbjg krb5kdc[3509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651980, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:40.250664 3522 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:19:40.251570 3522 webserver.cc:460] Webserver started at http://127.13.238.190:34439/ using document root <none> and password file <none>
I20260929 03:19:40.252135 3522 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:40.252205 3522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:40.252408 3522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:40.254524 3522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "758fb2d3a5d249ffb2f134a26339e12b"
format_stamp: "Formatted at 2026-09-29 03:19:40 on dist-test-slave-nbjg"
server_key: "1cfcaf3f79a0fc2a0c2b8d156d37b279"
server_key_iv: "10de10c5ae3434ec1b1b2a8b4e536ddd"
server_key_version: "encryptionkey@0"
I20260929 03:19:40.255062 3522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "758fb2d3a5d249ffb2f134a26339e12b"
format_stamp: "Formatted at 2026-09-29 03:19:40 on dist-test-slave-nbjg"
server_key: "1cfcaf3f79a0fc2a0c2b8d156d37b279"
server_key_iv: "10de10c5ae3434ec1b1b2a8b4e536ddd"
server_key_version: "encryptionkey@0"
I20260929 03:19:40.258816 3522 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 03:19:40.261171 3537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:40.262406 3522 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:19:40.262526 3522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "758fb2d3a5d249ffb2f134a26339e12b"
format_stamp: "Formatted at 2026-09-29 03:19:40 on dist-test-slave-nbjg"
server_key: "1cfcaf3f79a0fc2a0c2b8d156d37b279"
server_key_iv: "10de10c5ae3434ec1b1b2a8b4e536ddd"
server_key_version: "encryptionkey@0"
I20260929 03:19:40.262631 3522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:40.279248 3522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:40.282449 3522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:40.282634 3522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:40.289475 3522 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:36815
I20260929 03:19:40.289535 3589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:36815 every 8 connection(s)
I20260929 03:19:40.290508 3522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:40.292986 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 3522
I20260929 03:19:40.293145 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:19:40.293464 14266 external_mini_cluster.cc:1428] Setting key 36d68515538ad6002601a73f471d9853
I20260929 03:19:40.293569 3590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:40.300055 3590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 758fb2d3a5d249ffb2f134a26339e12b: Bootstrap starting.
I20260929 03:19:40.302381 3590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 758fb2d3a5d249ffb2f134a26339e12b: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:40.302564 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:40.294916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39475 (local address 127.13.238.190:36815)
0929 03:19:40.295393 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:40.295409 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:40.295447 (+ 38us) server_negotiation.cc:407] Connection header received
0929 03:19:40.296389 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:40.296409 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:40.296767 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:40.297106 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:40.297490 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:40.298534 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:40.299408 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:40.299876 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:40.300019 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:40.300912 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:40.300918 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:40.300931 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:40.301309 (+ 378us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:40.301358 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:40.301381 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:40.301491 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:40.301668 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 03:19:40.301796 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":161,"threads_started":1}
I20260929 03:19:40.303113 3590 log.cc:824] T 00000000000000000000000000000000 P 758fb2d3a5d249ffb2f134a26339e12b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:40.305199 3590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 758fb2d3a5d249ffb2f134a26339e12b: No bootstrap required, opened a new log
I20260929 03:19:40.305347 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:19:40.309167 3590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 758fb2d3a5d249ffb2f134a26339e12b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "758fb2d3a5d249ffb2f134a26339e12b" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36815 } }
I20260929 03:19:40.309417 3590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 758fb2d3a5d249ffb2f134a26339e12b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:40.309473 3590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 758fb2d3a5d249ffb2f134a26339e12b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 758fb2d3a5d249ffb2f134a26339e12b, State: Initialized, Role: FOLLOWER
I20260929 03:19:40.309924 3590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 758fb2d3a5d249ffb2f134a26339e12b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "758fb2d3a5d249ffb2f134a26339e12b" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36815 } }
I20260929 03:19:40.310078 3590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 758fb2d3a5d249ffb2f134a26339e12b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:40.310145 3590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 758fb2d3a5d249ffb2f134a26339e12b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:40.310263 3590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 758fb2d3a5d249ffb2f134a26339e12b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:40.310914 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:40.307185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60893 (local address 127.13.238.190:36815)
0929 03:19:40.307298 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:40.307302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:40.307397 (+ 95us) server_negotiation.cc:407] Connection header received
0929 03:19:40.307472 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:40.307475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:40.307513 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:40.307591 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:40.307795 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:40.308541 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:40.309276 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:40.309593 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:40.309691 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:40.310535 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:40.310539 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:40.310542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:40.310554 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:40.310581 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:40.310594 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:40.310676 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:40.310761 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 03:19:40.310792 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:19:40.311312 3590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 758fb2d3a5d249ffb2f134a26339e12b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "758fb2d3a5d249ffb2f134a26339e12b" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36815 } }
I20260929 03:19:40.311677 3590 leader_election.cc:302] T 00000000000000000000000000000000 P 758fb2d3a5d249ffb2f134a26339e12b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 758fb2d3a5d249ffb2f134a26339e12b; no voters:
I20260929 03:19:40.311993 3590 leader_election.cc:288] T 00000000000000000000000000000000 P 758fb2d3a5d249ffb2f134a26339e12b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:19:40.312058 3600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 758fb2d3a5d249ffb2f134a26339e12b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:40.312296 3600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 758fb2d3a5d249ffb2f134a26339e12b [term 1 LEADER]: Becoming Leader. State: Replica: 758fb2d3a5d249ffb2f134a26339e12b, State: Running, Role: LEADER
I20260929 03:19:40.312604 3600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 758fb2d3a5d249ffb2f134a26339e12b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "758fb2d3a5d249ffb2f134a26339e12b" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36815 } }
I20260929 03:19:40.313104 3590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 758fb2d3a5d249ffb2f134a26339e12b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:40.314136 3601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 758fb2d3a5d249ffb2f134a26339e12b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "758fb2d3a5d249ffb2f134a26339e12b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "758fb2d3a5d249ffb2f134a26339e12b" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36815 } } }
I20260929 03:19:40.314276 3601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 758fb2d3a5d249ffb2f134a26339e12b [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:40.314296 3602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 758fb2d3a5d249ffb2f134a26339e12b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 758fb2d3a5d249ffb2f134a26339e12b. Latest consensus state: current_term: 1 leader_uuid: "758fb2d3a5d249ffb2f134a26339e12b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "758fb2d3a5d249ffb2f134a26339e12b" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36815 } } }
I20260929 03:19:40.314385 3602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 758fb2d3a5d249ffb2f134a26339e12b [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:40.315274 3609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:40.318186 3609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:40.323674 3609 catalog_manager.cc:1347] Generated new cluster ID: 71b55dcd9bed411eb41d7a4882777cc8
I20260929 03:19:40.323738 3609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:40.348417 3609 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:40.349413 3609 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:40.356716 3609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 758fb2d3a5d249ffb2f134a26339e12b: Generated new TSK 0
I20260929 03:19:40.357435 3609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:19:40.366103 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 3522
W20260929 03:19:40.371135 3507 connection.cc:535] client connection to 127.13.238.190:36815 recv error: Network error: failed to read from TLS socket (remote: 127.13.238.190:36815): Connection reset by peer (error 104)
2026-09-29T03:19:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:19:40 dist-test-slave-nbjg krb5kdc[3621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:19:40 dist-test-slave-nbjg krb5kdc[3621](info): set up 2 sockets
Sep 29 03:19:40 dist-test-slave-nbjg krb5kdc[3621](info): commencing operation
krb5kdc: starting...
W20260929 03:19:41.318445 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:19:41 dist-test-slave-nbjg krb5kdc[3621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:19:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:19:41.437371 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:33223
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:40447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:33223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:41.546391 3634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:41.546687 3634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:41.546730 3634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:41.551071 3634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:41.551149 3634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:41.551169 3634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:41.551185 3634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:41.551200 3634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:41.556830 3634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:40447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:33223
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:33223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:41.558012 3634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:41.559146 3634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:41.567852 3640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:41.567852 3639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:41.567866 3642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:41.568089 3634 server_base.cc:1034] running on GCE node
I20260929 03:19:41.568789 3634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:41.569942 3634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:41.571115 3634 hybrid_clock.cc:648] HybridClock initialized: now 1790651981571075 us; error 49 us; skew 500 ppm
Sep 29 03:19:41 dist-test-slave-nbjg krb5kdc[3621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651981, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:41.573792 3634 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:19:41.574782 3634 webserver.cc:460] Webserver started at http://127.13.238.190:43767/ using document root <none> and password file <none>
I20260929 03:19:41.575343 3634 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:41.575408 3634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:41.575614 3634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:41.577728 3634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "7f010455e6f54b8bad594917491c60b0"
format_stamp: "Formatted at 2026-09-29 03:19:41 on dist-test-slave-nbjg"
server_key: "58123fbc2177f9da2109c8ecf9538ea1"
server_key_iv: "a67e4159140780825e5310e3d97547e6"
server_key_version: "encryptionkey@0"
I20260929 03:19:41.578327 3634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "7f010455e6f54b8bad594917491c60b0"
format_stamp: "Formatted at 2026-09-29 03:19:41 on dist-test-slave-nbjg"
server_key: "58123fbc2177f9da2109c8ecf9538ea1"
server_key_iv: "a67e4159140780825e5310e3d97547e6"
server_key_version: "encryptionkey@0"
I20260929 03:19:41.581928 3634 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:19:41.584383 3649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:41.585609 3634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:19:41.585721 3634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "7f010455e6f54b8bad594917491c60b0"
format_stamp: "Formatted at 2026-09-29 03:19:41 on dist-test-slave-nbjg"
server_key: "58123fbc2177f9da2109c8ecf9538ea1"
server_key_iv: "a67e4159140780825e5310e3d97547e6"
server_key_version: "encryptionkey@0"
I20260929 03:19:41.585811 3634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:41.590488 3634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:41.593446 3634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:41.593602 3634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:41.600317 3634 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:33223
I20260929 03:19:41.600332 3701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:33223 every 8 connection(s)
I20260929 03:19:41.601267 3634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:41.603276 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 3634
I20260929 03:19:41.603407 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:19:41.603663 14266 external_mini_cluster.cc:1428] Setting key 723815960b5dd3f00b23e2c6d379a48b
I20260929 03:19:41.604140 3702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:41.609086 3705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:41.605014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33135 (local address 127.13.238.190:33223)
0929 03:19:41.605464 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:41.605480 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:41.605739 (+ 259us) server_negotiation.cc:407] Connection header received
0929 03:19:41.606538 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:41.606556 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:41.606854 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:41.607126 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:41.607917 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:41.607922 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:41.607934 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:41.607944 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:41.607994 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:41.608013 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:41.608120 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:41.608226 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 03:19:41.608371 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":151,"threads_started":1}
I20260929 03:19:41.611037 3702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f010455e6f54b8bad594917491c60b0: Bootstrap starting.
I20260929 03:19:41.611827 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:19:41.613696 3702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f010455e6f54b8bad594917491c60b0: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:41.614549 3702 log.cc:824] T 00000000000000000000000000000000 P 7f010455e6f54b8bad594917491c60b0: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:41.615638 3705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:41.614018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44103 (local address 127.13.238.190:33223)
0929 03:19:41.614179 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:41.614184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:41.614413 (+ 229us) server_negotiation.cc:407] Connection header received
0929 03:19:41.614512 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:41.614515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:41.614547 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:41.614618 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:41.615291 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:41.615295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:41.615298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:41.615309 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:41.615334 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:41.615344 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:41.615408 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:41.615505 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 03:19:41.615534 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 03:19:41.616693 3702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f010455e6f54b8bad594917491c60b0: No bootstrap required, opened a new log
I20260929 03:19:41.620179 3702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f010455e6f54b8bad594917491c60b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f010455e6f54b8bad594917491c60b0" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 33223 } }
I20260929 03:19:41.620410 3702 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f010455e6f54b8bad594917491c60b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:41.620468 3702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f010455e6f54b8bad594917491c60b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f010455e6f54b8bad594917491c60b0, State: Initialized, Role: FOLLOWER
I20260929 03:19:41.620899 3702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f010455e6f54b8bad594917491c60b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f010455e6f54b8bad594917491c60b0" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 33223 } }
I20260929 03:19:41.621037 3702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f010455e6f54b8bad594917491c60b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:41.621101 3702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f010455e6f54b8bad594917491c60b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:41.621201 3702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f010455e6f54b8bad594917491c60b0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:41.622089 3702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f010455e6f54b8bad594917491c60b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f010455e6f54b8bad594917491c60b0" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 33223 } }
I20260929 03:19:41.622460 3702 leader_election.cc:302] T 00000000000000000000000000000000 P 7f010455e6f54b8bad594917491c60b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f010455e6f54b8bad594917491c60b0; no voters:
I20260929 03:19:41.622792 3702 leader_election.cc:288] T 00000000000000000000000000000000 P 7f010455e6f54b8bad594917491c60b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:19:41.622877 3712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f010455e6f54b8bad594917491c60b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:41.623051 3712 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f010455e6f54b8bad594917491c60b0 [term 1 LEADER]: Becoming Leader. State: Replica: 7f010455e6f54b8bad594917491c60b0, State: Running, Role: LEADER
I20260929 03:19:41.623351 3712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f010455e6f54b8bad594917491c60b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f010455e6f54b8bad594917491c60b0" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 33223 } }
I20260929 03:19:41.624161 3702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f010455e6f54b8bad594917491c60b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:41.624984 3713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f010455e6f54b8bad594917491c60b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f010455e6f54b8bad594917491c60b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f010455e6f54b8bad594917491c60b0" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 33223 } } }
I20260929 03:19:41.625114 3713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f010455e6f54b8bad594917491c60b0 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:41.625510 3714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f010455e6f54b8bad594917491c60b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f010455e6f54b8bad594917491c60b0. Latest consensus state: current_term: 1 leader_uuid: "7f010455e6f54b8bad594917491c60b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f010455e6f54b8bad594917491c60b0" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 33223 } } }
I20260929 03:19:41.625629 3714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f010455e6f54b8bad594917491c60b0 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:41.625495 3721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:41.629148 3721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:41.634373 3721 catalog_manager.cc:1347] Generated new cluster ID: a85276bf0abe44a583f2dc9a6a26dab8
I20260929 03:19:41.634434 3721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:41.650431 3721 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:41.650815 3721 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:41.663491 3721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f010455e6f54b8bad594917491c60b0: Generated new TSK 0
I20260929 03:19:41.664363 3721 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:19:41.671803 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 3634
2026-09-29T03:19:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:19:41 dist-test-slave-nbjg krb5kdc[3733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:19:41 dist-test-slave-nbjg krb5kdc[3733](info): set up 2 sockets
Sep 29 03:19:41 dist-test-slave-nbjg krb5kdc[3733](info): commencing operation
krb5kdc: starting...
W20260929 03:19:42.630466 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:19:42 dist-test-slave-nbjg krb5kdc[3733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:19:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:19:42.749259 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:33685
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:33685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:42.860894 3746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:42.861305 3746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:42.861383 3746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:42.866541 3746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:42.866608 3746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:42.866627 3746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:42.866647 3746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:42.866670 3746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:42.872288 3746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:33685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:33685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:42.873376 3746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:42.874456 3746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:42.882786 3754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:42.882932 3746 server_base.cc:1034] running on GCE node
W20260929 03:19:42.882802 3751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:42.882795 3752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:42.883605 3746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:42.884701 3746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:42.885850 3746 hybrid_clock.cc:648] HybridClock initialized: now 1790651982885833 us; error 30 us; skew 500 ppm
Sep 29 03:19:42 dist-test-slave-nbjg krb5kdc[3733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651982, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:42.891987 3746 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:19:42.892976 3746 webserver.cc:460] Webserver started at http://127.13.238.190:46169/ using document root <none> and password file <none>
I20260929 03:19:42.893549 3746 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:42.893613 3746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:42.893805 3746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:42.895956 3746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "ca8ed1e954f4413a817bc94e76fa9eec"
format_stamp: "Formatted at 2026-09-29 03:19:42 on dist-test-slave-nbjg"
server_key: "d4a68c65fe6cef30f5eb05ad22921cc9"
server_key_iv: "89f5a0585d524828cb0bd80f4c16a83b"
server_key_version: "encryptionkey@0"
I20260929 03:19:42.896517 3746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "ca8ed1e954f4413a817bc94e76fa9eec"
format_stamp: "Formatted at 2026-09-29 03:19:42 on dist-test-slave-nbjg"
server_key: "d4a68c65fe6cef30f5eb05ad22921cc9"
server_key_iv: "89f5a0585d524828cb0bd80f4c16a83b"
server_key_version: "encryptionkey@0"
I20260929 03:19:42.900444 3746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 03:19:42.902822 3761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:42.904069 3746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:19:42.904206 3746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "ca8ed1e954f4413a817bc94e76fa9eec"
format_stamp: "Formatted at 2026-09-29 03:19:42 on dist-test-slave-nbjg"
server_key: "d4a68c65fe6cef30f5eb05ad22921cc9"
server_key_iv: "89f5a0585d524828cb0bd80f4c16a83b"
server_key_version: "encryptionkey@0"
I20260929 03:19:42.904302 3746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:42.922072 3746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:42.925179 3746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:42.925348 3746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:42.932332 3746 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:33685
I20260929 03:19:42.932370 3813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:33685 every 8 connection(s)
I20260929 03:19:42.933295 3746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:42.935552 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 3746
I20260929 03:19:42.935700 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:19:42.936004 14266 external_mini_cluster.cc:1428] Setting key fe8ca64fd446c51adfc12f8708b836e3
I20260929 03:19:42.936335 3814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:42.943182 3814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca8ed1e954f4413a817bc94e76fa9eec: Bootstrap starting.
Sep 29 03:19:42 dist-test-slave-nbjg krb5kdc[3733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:19:42.945412 3814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca8ed1e954f4413a817bc94e76fa9eec: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:42.946228 3814 log.cc:824] T 00000000000000000000000000000000 P ca8ed1e954f4413a817bc94e76fa9eec: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:42.948024 3814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca8ed1e954f4413a817bc94e76fa9eec: No bootstrap required, opened a new log
I20260929 03:19:42.949420 3817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:42.937334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56039 (local address 127.13.238.190:33685)
0929 03:19:42.937733 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:42.937748 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:42.937783 (+ 35us) server_negotiation.cc:407] Connection header received
0929 03:19:42.938513 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:42.938532 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:42.938870 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:42.939150 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:42.939997 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:42.941001 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:42.942164 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:42.942657 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:42.942768 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:42.945142 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:42.945148 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:42.945168 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:42.945194 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:42.946814 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:42.947228 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:42.947233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:42.947237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:42.947343 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:42.947596 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:42.947599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:42.947600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:42.947982 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:42.948154 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:42.948455 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 03:19:42.948734 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":117,"threads_started":1}
I20260929 03:19:42.951941 3814 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca8ed1e954f4413a817bc94e76fa9eec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca8ed1e954f4413a817bc94e76fa9eec" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 33685 } }
I20260929 03:19:42.952207 3814 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca8ed1e954f4413a817bc94e76fa9eec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:42.952266 3814 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca8ed1e954f4413a817bc94e76fa9eec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8ed1e954f4413a817bc94e76fa9eec, State: Initialized, Role: FOLLOWER
I20260929 03:19:42.952332 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:19:42.952682 3814 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca8ed1e954f4413a817bc94e76fa9eec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca8ed1e954f4413a817bc94e76fa9eec" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 33685 } }
I20260929 03:19:42.952797 3814 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca8ed1e954f4413a817bc94e76fa9eec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:42.952857 3814 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca8ed1e954f4413a817bc94e76fa9eec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:42.952945 3814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca8ed1e954f4413a817bc94e76fa9eec [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:42.953872 3814 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca8ed1e954f4413a817bc94e76fa9eec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca8ed1e954f4413a817bc94e76fa9eec" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 33685 } }
I20260929 03:19:42.954190 3814 leader_election.cc:302] T 00000000000000000000000000000000 P ca8ed1e954f4413a817bc94e76fa9eec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca8ed1e954f4413a817bc94e76fa9eec; no voters:
I20260929 03:19:42.954525 3814 leader_election.cc:288] T 00000000000000000000000000000000 P ca8ed1e954f4413a817bc94e76fa9eec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:19:42.954582 3823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca8ed1e954f4413a817bc94e76fa9eec [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:42.955869 3814 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca8ed1e954f4413a817bc94e76fa9eec [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:42.956015 3823 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca8ed1e954f4413a817bc94e76fa9eec [term 1 LEADER]: Becoming Leader. State: Replica: ca8ed1e954f4413a817bc94e76fa9eec, State: Running, Role: LEADER
I20260929 03:19:42.956312 3823 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca8ed1e954f4413a817bc94e76fa9eec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca8ed1e954f4413a817bc94e76fa9eec" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 33685 } }
I20260929 03:19:42.958034 3825 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca8ed1e954f4413a817bc94e76fa9eec [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca8ed1e954f4413a817bc94e76fa9eec. Latest consensus state: current_term: 1 leader_uuid: "ca8ed1e954f4413a817bc94e76fa9eec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca8ed1e954f4413a817bc94e76fa9eec" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 33685 } } }
I20260929 03:19:42.958316 3826 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca8ed1e954f4413a817bc94e76fa9eec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca8ed1e954f4413a817bc94e76fa9eec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca8ed1e954f4413a817bc94e76fa9eec" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 33685 } } }
I20260929 03:19:42.958401 3826 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca8ed1e954f4413a817bc94e76fa9eec [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:42.958171 3825 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca8ed1e954f4413a817bc94e76fa9eec [sys.catalog]: This master's current role is: LEADER
W20260929 03:19:42.960706 3838 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca8ed1e954f4413a817bc94e76fa9eec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:19:42.960780 3838 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:19:42.960857 3839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:42.962077 3839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:42.962782 3817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:42.954479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34291 (local address 127.13.238.190:33685)
0929 03:19:42.954677 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:42.954682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:42.954901 (+ 219us) server_negotiation.cc:407] Connection header received
0929 03:19:42.955010 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:42.955013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:42.955056 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:42.955125 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:42.955857 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:42.956552 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:42.957475 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:42.957815 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:42.957897 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:42.959719 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:42.959722 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:42.959725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:42.959749 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:42.961226 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:42.961614 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:42.961617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:42.961619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:42.961697 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:42.961930 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:42.961933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:42.961935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:42.962099 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:42.962232 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:42.962534 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 03:19:42.962638 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260929 03:19:42.967368 3839 catalog_manager.cc:1347] Generated new cluster ID: 092d7769e58c400c83bf9db6851d5f30
I20260929 03:19:42.967435 3839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:42.982247 3839 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:42.983562 3839 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:42.988714 3839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca8ed1e954f4413a817bc94e76fa9eec: Generated new TSK 0
I20260929 03:19:42.989365 3839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:19:42.998984 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 3746
2026-09-29T03:19:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:19:43 dist-test-slave-nbjg krb5kdc[3845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:19:43 dist-test-slave-nbjg krb5kdc[3845](info): set up 2 sockets
Sep 29 03:19:43 dist-test-slave-nbjg krb5kdc[3845](info): commencing operation
krb5kdc: starting...
W20260929 03:19:43.958564 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:19:44 dist-test-slave-nbjg krb5kdc[3845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:19:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:19:44.077306 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:42989
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:42989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:44.188831 3858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:44.189129 3858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:44.189198 3858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:44.193315 3858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:44.193383 3858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:44.193410 3858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:44.193430 3858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:44.193451 3858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:44.198966 3858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:42989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:42989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:44.200115 3858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:44.201363 3858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:44.211791 3866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:44.211814 3863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:44.212064 3858 server_base.cc:1034] running on GCE node
W20260929 03:19:44.211784 3864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:44.212967 3858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:44.214069 3858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:44.215250 3858 hybrid_clock.cc:648] HybridClock initialized: now 1790651984215221 us; error 40 us; skew 500 ppm
Sep 29 03:19:44 dist-test-slave-nbjg krb5kdc[3845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651984, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:44.218640 3858 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:19:44.219712 3858 webserver.cc:460] Webserver started at http://127.13.238.190:35103/ using document root <none> and password file <none>
I20260929 03:19:44.220413 3858 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:44.220498 3858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:44.220759 3858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:44.223703 3858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "d94d4f2991de452db5e72d6c73aee507"
format_stamp: "Formatted at 2026-09-29 03:19:44 on dist-test-slave-nbjg"
server_key: "20020a1850e1ff14cce129498ee4031d"
server_key_iv: "1d59dcece52e0986991c5a5129be0bdf"
server_key_version: "encryptionkey@0"
I20260929 03:19:44.224407 3858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "d94d4f2991de452db5e72d6c73aee507"
format_stamp: "Formatted at 2026-09-29 03:19:44 on dist-test-slave-nbjg"
server_key: "20020a1850e1ff14cce129498ee4031d"
server_key_iv: "1d59dcece52e0986991c5a5129be0bdf"
server_key_version: "encryptionkey@0"
I20260929 03:19:44.229014 3858 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 03:19:44.231446 3873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:44.232689 3858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 03:19:44.232800 3858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "d94d4f2991de452db5e72d6c73aee507"
format_stamp: "Formatted at 2026-09-29 03:19:44 on dist-test-slave-nbjg"
server_key: "20020a1850e1ff14cce129498ee4031d"
server_key_iv: "1d59dcece52e0986991c5a5129be0bdf"
server_key_version: "encryptionkey@0"
I20260929 03:19:44.232916 3858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:44.251060 3858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:44.254029 3858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:44.254238 3858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:44.261013 3858 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:42989
I20260929 03:19:44.261044 3925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:42989 every 8 connection(s)
I20260929 03:19:44.261978 3858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:44.263612 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 3858
I20260929 03:19:44.263731 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:19:44.264005 14266 external_mini_cluster.cc:1428] Setting key 0a2820327acbd53ee6cb0363a4ce2937
I20260929 03:19:44.264957 3926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:44.271660 3926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d94d4f2991de452db5e72d6c73aee507: Bootstrap starting.
I20260929 03:19:44.274044 3929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:44.265426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42627 (local address 127.13.238.190:42989)
0929 03:19:44.265850 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:44.265868 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:44.265910 (+ 42us) server_negotiation.cc:407] Connection header received
0929 03:19:44.266766 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:44.266791 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:44.267232 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:44.267593 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:44.267988 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:44.269443 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:44.270412 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:44.271112 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:44.271272 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:44.272018 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:44.272024 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:44.272040 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:44.272521 (+ 481us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:44.272588 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:44.272617 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:44.272714 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:44.272940 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 03:19:44.273145 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":123,"threads_started":1}
I20260929 03:19:44.274312 3926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d94d4f2991de452db5e72d6c73aee507: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:44.275025 3926 log.cc:824] T 00000000000000000000000000000000 P d94d4f2991de452db5e72d6c73aee507: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:44.276793 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:19:44.276939 3926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d94d4f2991de452db5e72d6c73aee507: No bootstrap required, opened a new log
I20260929 03:19:44.281045 3926 raft_consensus.cc:348] T 00000000000000000000000000000000 P d94d4f2991de452db5e72d6c73aee507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94d4f2991de452db5e72d6c73aee507" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42989 } }
I20260929 03:19:44.281280 3926 raft_consensus.cc:374] T 00000000000000000000000000000000 P d94d4f2991de452db5e72d6c73aee507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:44.281340 3926 raft_consensus.cc:729] T 00000000000000000000000000000000 P d94d4f2991de452db5e72d6c73aee507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d94d4f2991de452db5e72d6c73aee507, State: Initialized, Role: FOLLOWER
I20260929 03:19:44.281790 3926 consensus_queue.cc:260] T 00000000000000000000000000000000 P d94d4f2991de452db5e72d6c73aee507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94d4f2991de452db5e72d6c73aee507" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42989 } }
I20260929 03:19:44.281939 3926 raft_consensus.cc:388] T 00000000000000000000000000000000 P d94d4f2991de452db5e72d6c73aee507 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:44.281999 3926 raft_consensus.cc:482] T 00000000000000000000000000000000 P d94d4f2991de452db5e72d6c73aee507 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:44.282081 3926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d94d4f2991de452db5e72d6c73aee507 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:44.282748 3929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:44.278945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52099 (local address 127.13.238.190:42989)
0929 03:19:44.279053 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:44.279057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:44.279284 (+ 227us) server_negotiation.cc:407] Connection header received
0929 03:19:44.279372 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:44.279376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:44.279423 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:44.279505 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:44.279709 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:44.280471 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:44.281219 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:44.281551 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:44.281646 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:44.282400 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:44.282404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:44.282406 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:44.282418 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:44.282443 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:44.282455 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:44.282528 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:44.282580 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 03:19:44.282627 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:19:44.283149 3926 raft_consensus.cc:504] T 00000000000000000000000000000000 P d94d4f2991de452db5e72d6c73aee507 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94d4f2991de452db5e72d6c73aee507" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42989 } }
I20260929 03:19:44.283535 3926 leader_election.cc:302] T 00000000000000000000000000000000 P d94d4f2991de452db5e72d6c73aee507 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d94d4f2991de452db5e72d6c73aee507; no voters:
I20260929 03:19:44.283856 3926 leader_election.cc:288] T 00000000000000000000000000000000 P d94d4f2991de452db5e72d6c73aee507 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:19:44.283934 3936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d94d4f2991de452db5e72d6c73aee507 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:44.284130 3936 raft_consensus.cc:686] T 00000000000000000000000000000000 P d94d4f2991de452db5e72d6c73aee507 [term 1 LEADER]: Becoming Leader. State: Replica: d94d4f2991de452db5e72d6c73aee507, State: Running, Role: LEADER
I20260929 03:19:44.284411 3936 consensus_queue.cc:237] T 00000000000000000000000000000000 P d94d4f2991de452db5e72d6c73aee507 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94d4f2991de452db5e72d6c73aee507" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42989 } }
I20260929 03:19:44.284960 3926 sys_catalog.cc:564] T 00000000000000000000000000000000 P d94d4f2991de452db5e72d6c73aee507 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:44.285948 3937 sys_catalog.cc:455] T 00000000000000000000000000000000 P d94d4f2991de452db5e72d6c73aee507 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d94d4f2991de452db5e72d6c73aee507" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94d4f2991de452db5e72d6c73aee507" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42989 } } }
I20260929 03:19:44.286072 3937 sys_catalog.cc:458] T 00000000000000000000000000000000 P d94d4f2991de452db5e72d6c73aee507 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:44.286322 3938 sys_catalog.cc:455] T 00000000000000000000000000000000 P d94d4f2991de452db5e72d6c73aee507 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d94d4f2991de452db5e72d6c73aee507. Latest consensus state: current_term: 1 leader_uuid: "d94d4f2991de452db5e72d6c73aee507" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94d4f2991de452db5e72d6c73aee507" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42989 } } }
I20260929 03:19:44.286461 3938 sys_catalog.cc:458] T 00000000000000000000000000000000 P d94d4f2991de452db5e72d6c73aee507 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:44.287154 3945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:44.289868 3945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:44.295312 3945 catalog_manager.cc:1347] Generated new cluster ID: 9c83dab428cf4ac59b648d779ae7b866
I20260929 03:19:44.295377 3945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:44.320465 3945 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:44.321767 3945 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:44.329167 3945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d94d4f2991de452db5e72d6c73aee507: Generated new TSK 0
I20260929 03:19:44.329882 3945 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:19:44.340934 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 3858
2026-09-29T03:19:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:19:44 dist-test-slave-nbjg krb5kdc[3957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:19:44 dist-test-slave-nbjg krb5kdc[3957](info): set up 2 sockets
Sep 29 03:19:44 dist-test-slave-nbjg krb5kdc[3957](info): commencing operation
krb5kdc: starting...
W20260929 03:19:45.290591 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:19:45 dist-test-slave-nbjg krb5kdc[3957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:19:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:19:45.409108 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:36247
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:33139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:36247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:45.520385 3970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:45.520687 3970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:45.520749 3970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:45.524876 3970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:45.524945 3970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:45.524971 3970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:45.524999 3970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:45.525020 3970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:45.530431 3970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:33139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:36247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:36247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:45.531826 3970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:45.533088 3970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:45.541769 3975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:45.541805 3970 server_base.cc:1034] running on GCE node
W20260929 03:19:45.541791 3978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:45.541769 3976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:45.542616 3970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:45.543716 3970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:45.544881 3970 hybrid_clock.cc:648] HybridClock initialized: now 1790651985544862 us; error 28 us; skew 500 ppm
Sep 29 03:19:45 dist-test-slave-nbjg krb5kdc[3957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651985, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:45.547598 3970 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:19:45.548451 3970 webserver.cc:460] Webserver started at http://127.13.238.190:41535/ using document root <none> and password file <none>
I20260929 03:19:45.549044 3970 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:45.549109 3970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:45.549279 3970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:45.551386 3970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "e00dce65c2374d1d911ce9a972d361a1"
format_stamp: "Formatted at 2026-09-29 03:19:45 on dist-test-slave-nbjg"
server_key: "bc2879f998dbb5ed11cb54b5fbfec665"
server_key_iv: "5ec15e0c274c1f60d4fdc246e2b5baaf"
server_key_version: "encryptionkey@0"
I20260929 03:19:45.551934 3970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "e00dce65c2374d1d911ce9a972d361a1"
format_stamp: "Formatted at 2026-09-29 03:19:45 on dist-test-slave-nbjg"
server_key: "bc2879f998dbb5ed11cb54b5fbfec665"
server_key_iv: "5ec15e0c274c1f60d4fdc246e2b5baaf"
server_key_version: "encryptionkey@0"
I20260929 03:19:45.555747 3970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:19:45.558547 3985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:45.559923 3970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:19:45.560046 3970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "e00dce65c2374d1d911ce9a972d361a1"
format_stamp: "Formatted at 2026-09-29 03:19:45 on dist-test-slave-nbjg"
server_key: "bc2879f998dbb5ed11cb54b5fbfec665"
server_key_iv: "5ec15e0c274c1f60d4fdc246e2b5baaf"
server_key_version: "encryptionkey@0"
I20260929 03:19:45.560148 3970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:45.565081 3970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:45.568222 3970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:45.568384 3970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:45.575284 3970 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:36247
I20260929 03:19:45.575304 4037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:36247 every 8 connection(s)
I20260929 03:19:45.576258 3970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:45.579305 4038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:45.585242 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 3970
I20260929 03:19:45.585390 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:19:45.585361 4038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e00dce65c2374d1d911ce9a972d361a1: Bootstrap starting.
I20260929 03:19:45.585713 14266 external_mini_cluster.cc:1428] Setting key 960253d3b2f19fc73be17e9fd1d4ec4f
I20260929 03:19:45.588258 4038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e00dce65c2374d1d911ce9a972d361a1: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:45.589000 4038 log.cc:824] T 00000000000000000000000000000000 P e00dce65c2374d1d911ce9a972d361a1: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:45.591025 4038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e00dce65c2374d1d911ce9a972d361a1: No bootstrap required, opened a new log
I20260929 03:19:45.591082 4041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:45.587196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42657 (local address 127.13.238.190:36247)
0929 03:19:45.587699 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:45.587714 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:45.587752 (+ 38us) server_negotiation.cc:407] Connection header received
0929 03:19:45.588455 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:45.588475 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:45.588781 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:45.589099 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:45.589838 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:45.589843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:45.589857 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:45.589867 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:45.589922 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:45.589941 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:45.590034 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:45.590168 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 03:19:45.590356 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":146,"threads_started":1}
I20260929 03:19:45.594015 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:19:45.595038 4038 raft_consensus.cc:348] T 00000000000000000000000000000000 P e00dce65c2374d1d911ce9a972d361a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e00dce65c2374d1d911ce9a972d361a1" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36247 } }
I20260929 03:19:45.595284 4038 raft_consensus.cc:374] T 00000000000000000000000000000000 P e00dce65c2374d1d911ce9a972d361a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:19:45.595357 4038 raft_consensus.cc:729] T 00000000000000000000000000000000 P e00dce65c2374d1d911ce9a972d361a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e00dce65c2374d1d911ce9a972d361a1, State: Initialized, Role: FOLLOWER
I20260929 03:19:45.595804 4038 consensus_queue.cc:260] T 00000000000000000000000000000000 P e00dce65c2374d1d911ce9a972d361a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e00dce65c2374d1d911ce9a972d361a1" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36247 } }
I20260929 03:19:45.595940 4038 raft_consensus.cc:388] T 00000000000000000000000000000000 P e00dce65c2374d1d911ce9a972d361a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:19:45.596005 4038 raft_consensus.cc:482] T 00000000000000000000000000000000 P e00dce65c2374d1d911ce9a972d361a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:19:45.596107 4038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e00dce65c2374d1d911ce9a972d361a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:45.597256 4038 raft_consensus.cc:504] T 00000000000000000000000000000000 P e00dce65c2374d1d911ce9a972d361a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e00dce65c2374d1d911ce9a972d361a1" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36247 } }
I20260929 03:19:45.597625 4038 leader_election.cc:302] T 00000000000000000000000000000000 P e00dce65c2374d1d911ce9a972d361a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e00dce65c2374d1d911ce9a972d361a1; no voters:
I20260929 03:19:45.597890 4041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:45.596229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54741 (local address 127.13.238.190:36247)
0929 03:19:45.596356 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:45.596360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:45.596524 (+ 164us) server_negotiation.cc:407] Connection header received
0929 03:19:45.596634 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:45.596637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:45.596701 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:45.596774 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:45.597528 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:45.597531 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:45.597534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:19:45.597545 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:45.597574 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:19:45.597586 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:45.597661 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:45.597712 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 03:19:45.597751 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:19:45.597976 4038 leader_election.cc:288] T 00000000000000000000000000000000 P e00dce65c2374d1d911ce9a972d361a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:19:45.598306 4049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e00dce65c2374d1d911ce9a972d361a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:45.598564 4049 raft_consensus.cc:686] T 00000000000000000000000000000000 P e00dce65c2374d1d911ce9a972d361a1 [term 1 LEADER]: Becoming Leader. State: Replica: e00dce65c2374d1d911ce9a972d361a1, State: Running, Role: LEADER
I20260929 03:19:45.598935 4049 consensus_queue.cc:237] T 00000000000000000000000000000000 P e00dce65c2374d1d911ce9a972d361a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e00dce65c2374d1d911ce9a972d361a1" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36247 } }
I20260929 03:19:45.599313 4038 sys_catalog.cc:564] T 00000000000000000000000000000000 P e00dce65c2374d1d911ce9a972d361a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:45.600634 4050 sys_catalog.cc:455] T 00000000000000000000000000000000 P e00dce65c2374d1d911ce9a972d361a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e00dce65c2374d1d911ce9a972d361a1. Latest consensus state: current_term: 1 leader_uuid: "e00dce65c2374d1d911ce9a972d361a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e00dce65c2374d1d911ce9a972d361a1" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36247 } } }
I20260929 03:19:45.600776 4050 sys_catalog.cc:458] T 00000000000000000000000000000000 P e00dce65c2374d1d911ce9a972d361a1 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:45.600642 4048 sys_catalog.cc:455] T 00000000000000000000000000000000 P e00dce65c2374d1d911ce9a972d361a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e00dce65c2374d1d911ce9a972d361a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e00dce65c2374d1d911ce9a972d361a1" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36247 } } }
I20260929 03:19:45.601039 4048 sys_catalog.cc:458] T 00000000000000000000000000000000 P e00dce65c2374d1d911ce9a972d361a1 [sys.catalog]: This master's current role is: LEADER
W20260929 03:19:45.603861 4063 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e00dce65c2374d1d911ce9a972d361a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:19:45.603964 4063 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:19:45.604070 4061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:45.604789 4061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:45.610350 4061 catalog_manager.cc:1347] Generated new cluster ID: 25939dd70d1046bbbdb6399238c36bd1
I20260929 03:19:45.610415 4061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:45.628882 4061 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:45.629276 4061 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:45.642793 4061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e00dce65c2374d1d911ce9a972d361a1: Generated new TSK 0
I20260929 03:19:45.643587 4061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:19:45.648207 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 3970
2026-09-29T03:19:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1306 ms)
[----------] 18 tests from AuthTokenIssuingTest (23756 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:19:45 dist-test-slave-nbjg krb5kdc[4069](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:19:45 dist-test-slave-nbjg krb5kdc[4069](info): set up 2 sockets
Sep 29 03:19:45 dist-test-slave-nbjg krb5kdc[4069](info): commencing operation
krb5kdc: starting...
W20260929 03:19:46.606439 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:19:46 dist-test-slave-nbjg krb5kdc[4069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:19:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:19:46.723074 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:42551
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:46793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:42551,127.13.238.189:39667,127.13.238.188:36077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:46.833683 4082 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:19:46.833964 4082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:46.833986 4082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:46.834033 4082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:46.838215 4082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:46.838279 4082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:46.838297 4082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:46.838312 4082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:46.838328 4082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:46.843683 4082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:46793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:42551,127.13.238.189:39667,127.13.238.188:36077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:42551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:46.844934 4082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:46.846042 4082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:46.854347 4090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:46.854346 4087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:46.854346 4088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:46.854382 4082 server_base.cc:1034] running on GCE node
I20260929 03:19:46.855126 4082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:46.856197 4082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:46.857376 4082 hybrid_clock.cc:648] HybridClock initialized: now 1790651986857357 us; error 29 us; skew 500 ppm
Sep 29 03:19:46 dist-test-slave-nbjg krb5kdc[4069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651986, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:46.860054 4082 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:19:46.860908 4082 webserver.cc:460] Webserver started at http://127.13.238.190:43073/ using document root <none> and password file <none>
I20260929 03:19:46.861431 4082 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:46.861493 4082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:46.861701 4082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:46.863758 4082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "7972ed36fa3542cf8f7e7c15680a214e"
format_stamp: "Formatted at 2026-09-29 03:19:46 on dist-test-slave-nbjg"
server_key: "05043f9b954bca695917ea314989c343"
server_key_iv: "062f0e86a615edeb03fb32bc284e8584"
server_key_version: "encryptionkey@0"
I20260929 03:19:46.864283 4082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "7972ed36fa3542cf8f7e7c15680a214e"
format_stamp: "Formatted at 2026-09-29 03:19:46 on dist-test-slave-nbjg"
server_key: "05043f9b954bca695917ea314989c343"
server_key_iv: "062f0e86a615edeb03fb32bc284e8584"
server_key_version: "encryptionkey@0"
I20260929 03:19:46.867977 4082 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 03:19:46.870467 4097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:46.871739 4082 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 03:19:46.871861 4082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "7972ed36fa3542cf8f7e7c15680a214e"
format_stamp: "Formatted at 2026-09-29 03:19:46 on dist-test-slave-nbjg"
server_key: "05043f9b954bca695917ea314989c343"
server_key_iv: "062f0e86a615edeb03fb32bc284e8584"
server_key_version: "encryptionkey@0"
I20260929 03:19:46.871959 4082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:46.888403 4082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:46.891165 4082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:46.891337 4082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:46.897871 4082 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:42551
I20260929 03:19:46.898130 4149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:42551 every 8 connection(s)
I20260929 03:19:46.898861 4082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:46.899075 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 4082
I20260929 03:19:46.899214 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:19:46.899523 14266 external_mini_cluster.cc:1428] Setting key 2f2e15b1bf61e043733dc01b63a3e969
I20260929 03:19:46.902066 4150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:46.905877 4150 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42551 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.238.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.189@KRBTEST.COM" created.
Sep 29 03:19:46 dist-test-slave-nbjg krb5kdc[4069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651986, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:19:46.921602 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:46.909072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40856 (local address 127.13.238.190:42551)
0929 03:19:46.909407 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:46.909415 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:46.909452 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:19:46.910041 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:46.910046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:46.910321 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:46.910510 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:46.911475 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:46.912393 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:46.913841 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:46.914271 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:46.914389 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:46.917127 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:46.917133 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:46.917143 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:46.917165 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:46.918754 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:46.919213 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:46.919218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:46.919219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:46.919322 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:46.919631 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:46.919633 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:46.919635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:46.920083 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:46.920182 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:46.920574 (+ 392us) server_negotiation.cc:299] Negotiation successful
0929 03:19:46.920744 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":72,"threads_started":1}
I20260929 03:19:46.921620 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:46.908182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42551 (local address 127.0.0.1:40856)
0929 03:19:46.908720 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:46.908767 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:46.909727 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:46.910507 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:46.910512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:46.910921 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:46.911285 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:46.911304 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:46.912520 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:46.912524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:46.913716 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:46.913729 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:46.914418 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:46.914423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:46.914558 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:46.915168 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:46.915190 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:46.916966 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:46.918859 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:46.918868 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:46.918877 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:46.919104 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:46.919419 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:46.919423 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:46.919425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:46.919545 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:46.920203 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:46.920209 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:46.920338 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 03:19:46.920479 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 03:19:46.920809 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":60,"thread_start_us":137,"threads_started":1}
I20260929 03:19:46.924007 4150 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.189" port: 39667 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.189.keytab.
Entry for principal kudu/127.13.238.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.189.keytab.
I20260929 03:19:46.924762 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:46.924485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:39667 (local address 127.0.0.1:60488)
0929 03:19:46.924606 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:46.924693 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.189:39667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 03:19:46.924952 4101 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.238.189:39667: connect: Connection refused (error 111)
I20260929 03:19:46.925707 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.189:39667
--webserver_interface=127.13.238.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.189.keytab
--principal=kudu/127.13.238.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:46793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:42551,127.13.238.189:39667,127.13.238.188:36077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:19:46.925760 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:46.925552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:39667 (local address 127.0.0.1:60502)
0929 03:19:46.925654 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:46.925704 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.189:39667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 03:19:46.925894 4150 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.189:39667: Network error: Client connection negotiation failed: client connection to 127.13.238.189:39667: connect: Connection refused (error 111)
I20260929 03:19:46.975111 4150 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.189" port: 39667 } attempt: 1
I20260929 03:19:46.975903 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:46.975609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:39667 (local address 127.0.0.1:60518)
0929 03:19:46.975754 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:46.975822 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.189:39667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 03:19:46.976764 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:46.976530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:39667 (local address 127.0.0.1:60528)
0929 03:19:46.976650 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:46.976701 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.189:39667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 03:19:46.976908 4150 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.189:39667: Network error: Client connection negotiation failed: client connection to 127.13.238.189:39667: connect: Connection refused (error 111)
W20260929 03:19:47.035881 4157 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:19:47.036244 4157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:47.036288 4157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:47.036350 4157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:47.040401 4157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:47.040462 4157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:47.040488 4157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:47.040508 4157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:47.040532 4157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:47.045089 4150 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.189" port: 39667 } attempt: 2
I20260929 03:19:47.045943 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.045645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:39667 (local address 127.0.0.1:60532)
0929 03:19:47.045797 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:47.045863 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.189:39667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 03:19:47.046060 4157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:46793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:42551,127.13.238.189:39667,127.13.238.188:36077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.189.keytab
--principal=kudu/127.13.238.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.189:39667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.238.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:47.046804 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.046603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:39667 (local address 127.0.0.1:60536)
0929 03:19:47.046698 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:47.046746 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.189:39667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 03:19:47.046945 4150 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.189:39667: Network error: Client connection negotiation failed: client connection to 127.13.238.189:39667: connect: Connection refused (error 111)
I20260929 03:19:47.047356 4157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:47.048355 4157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:47.056447 4165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:47.056504 4163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:47.056458 4162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:47.056928 4157 server_base.cc:1034] running on GCE node
I20260929 03:19:47.057518 4157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:47.058622 4157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:47.059813 4157 hybrid_clock.cc:648] HybridClock initialized: now 1790651987059790 us; error 29 us; skew 500 ppm
Sep 29 03:19:47 dist-test-slave-nbjg krb5kdc[4069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651987, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:47.062477 4157 init.cc:377] Logged in from keytab as kudu/127.13.238.189@KRBTEST.COM (short username kudu)
I20260929 03:19:47.063334 4157 webserver.cc:460] Webserver started at http://127.13.238.189:36833/ using document root <none> and password file <none>
I20260929 03:19:47.063856 4157 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:47.063916 4157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:47.064144 4157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:47.066111 4157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-1/data/instance:
uuid: "18a1861f6de748928118624338c0bcf0"
format_stamp: "Formatted at 2026-09-29 03:19:47 on dist-test-slave-nbjg"
server_key: "c2abfa3d84a4c3879578d3042af3e5ef"
server_key_iv: "6ca55ceabd673d912fd414277dbacf49"
server_key_version: "encryptionkey@0"
I20260929 03:19:47.066677 4157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-1/wal/instance:
uuid: "18a1861f6de748928118624338c0bcf0"
format_stamp: "Formatted at 2026-09-29 03:19:47 on dist-test-slave-nbjg"
server_key: "c2abfa3d84a4c3879578d3042af3e5ef"
server_key_iv: "6ca55ceabd673d912fd414277dbacf49"
server_key_version: "encryptionkey@0"
I20260929 03:19:47.070254 4157 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:19:47.072618 4172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:47.073889 4157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 03:19:47.074018 4157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-1/wal
uuid: "18a1861f6de748928118624338c0bcf0"
format_stamp: "Formatted at 2026-09-29 03:19:47 on dist-test-slave-nbjg"
server_key: "c2abfa3d84a4c3879578d3042af3e5ef"
server_key_iv: "6ca55ceabd673d912fd414277dbacf49"
server_key_version: "encryptionkey@0"
I20260929 03:19:47.074169 4157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:47.091862 4157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:47.094695 4157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:47.094872 4157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:47.102180 4157 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.189:39667
I20260929 03:19:47.102459 4224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.189:39667 every 8 connection(s)
I20260929 03:19:47.103186 4157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-1/data/info.pb
I20260929 03:19:47.105931 4225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:47.109642 4225 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42551 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:19:47.111893 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 4157
I20260929 03:19:47.112010 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-1/wal/instance
I20260929 03:19:47.112262 14266 external_mini_cluster.cc:1428] Setting key e881d017ae8ee9adbf52f92e00d9cfc5
Sep 29 03:19:47 dist-test-slave-nbjg krb5kdc[4069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651987, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.189@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.238.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.188@KRBTEST.COM" created.
I20260929 03:19:47.124078 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.111552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40872 (local address 127.13.238.190:42551)
0929 03:19:47.111706 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:47.111711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:47.112576 (+ 865us) server_negotiation.cc:407] Connection header received
0929 03:19:47.113558 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:47.113562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:47.113621 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:47.113693 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:47.114954 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.115487 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.116616 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.116861 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.116951 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:47.119793 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:47.119797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:47.119799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:47.119829 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:47.121725 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.122387 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.122391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.122392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.122473 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.122841 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.122843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.122844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.123019 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:47.123088 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:47.123758 (+ 670us) server_negotiation.cc:299] Negotiation successful
0929 03:19:47.123900 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:19:47.124827 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.111921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42551 (local address 127.0.0.1:40872)
0929 03:19:47.112414 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:47.112467 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:47.113332 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:47.113879 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:47.113886 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:47.114299 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:47.114801 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:47.114816 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.115624 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.115627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:47.116489 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:47.116499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.117031 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.117036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:47.117200 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:47.117807 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:47.117826 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:47.119582 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:47.121853 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:47.121859 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:47.121874 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:47.122277 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:47.122556 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:47.122559 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:47.122561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:47.122763 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:47.123102 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:47.123108 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:47.123476 (+ 368us) client_negotiation.cc:769] Sending connection context
0929 03:19:47.123643 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 03:19:47.123920 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":131,"threads_started":1}
I20260929 03:19:47.126483 4225 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.189" port: 39667 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:19:47 dist-test-slave-nbjg krb5kdc[4069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651987, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.189@KRBTEST.COM for kudu/127.13.238.189@KRBTEST.COM
I20260929 03:19:47.134948 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.126965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:39667 (local address 127.0.0.1:60542)
0929 03:19:47.127117 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:47.127128 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:47.127187 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:47.127721 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:47.127724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:47.127879 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:47.127955 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:47.127960 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.128903 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.128906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:47.129421 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:47.129428 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.129972 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.129976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:47.130055 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:47.130474 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:47.130486 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:47.131817 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:47.133426 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:47.133429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:47.133431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:47.133603 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:47.133869 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:47.133871 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:47.133873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:47.133947 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:47.134449 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:47.134452 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:47.134543 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 03:19:47.134631 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 03:19:47.134784 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 03:19:47.135052 4230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.127077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60542 (local address 127.13.238.189:39667)
0929 03:19:47.127390 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:47.127398 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:47.127416 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:19:47.127447 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:47.127451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:47.127602 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:47.127718 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:47.128050 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.128794 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.129519 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.129873 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.129960 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:47.131931 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:47.131940 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:47.131952 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:47.131971 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:47.133332 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.133695 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.133700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.133701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.133770 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.134019 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.134022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.134024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.134357 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:47.134477 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:47.134766 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 03:19:47.134894 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":133,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.188.keytab.
Entry for principal kudu/127.13.238.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.188.keytab.
I20260929 03:19:47.137152 4225 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:19:47.137374 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.188:36077
--webserver_interface=127.13.238.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.188.keytab
--principal=kudu/127.13.238.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:46793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:42551,127.13.238.189:39667,127.13.238.188:36077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:19:47.137854 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.137604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:36077 (local address 127.0.0.1:59532)
0929 03:19:47.137717 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:47.137789 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:36077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 03:19:47.138062 4175 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.238.188:36077: connect: Connection refused (error 111)
I20260929 03:19:47.138110 4150 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.189" port: 39667 } attempt: 3
I20260929 03:19:47.139258 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.139039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:36077 (local address 127.0.0.1:59538)
0929 03:19:47.139139 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:47.139189 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:36077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 03:19:47.139459 4225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.188:36077: Network error: Client connection negotiation failed: client connection to 127.13.238.188:36077: connect: Connection refused (error 111)
Sep 29 03:19:47 dist-test-slave-nbjg krb5kdc[4069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651986, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for kudu/127.13.238.189@KRBTEST.COM
I20260929 03:19:47.146580 4230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.138778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60550 (local address 127.13.238.189:39667)
0929 03:19:47.138877 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:47.138881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:47.138895 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:19:47.139008 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:47.139011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:47.139048 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:47.139575 (+ 527us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:47.139657 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.140131 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.140953 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.141608 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.141698 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:47.143705 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:47.143708 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:47.143710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:47.143732 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:47.144984 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.145435 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.145437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.145439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.145505 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.145766 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.145768 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.145769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.145917 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:47.145982 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:47.146373 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 03:19:47.146468 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:19:47.146653 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.138553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:39667 (local address 127.0.0.1:60550)
0929 03:19:47.138806 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:47.138821 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:47.138903 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:47.139155 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:47.139158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:47.139339 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:47.139467 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:47.139476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.140236 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.140240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:47.140851 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:47.140861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.141723 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.141727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:47.141836 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:47.142274 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:47.142288 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:47.143559 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:47.145089 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:47.145093 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:47.145095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:47.145348 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:47.145586 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:47.145589 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:47.145591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:47.145688 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:47.146005 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:47.146009 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:47.146123 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 03:19:47.146326 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 03:19:47.146482 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260929 03:19:47.147418 4150 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:19:47.148080 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.147829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:36077 (local address 127.0.0.1:59540)
0929 03:19:47.147947 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:47.148011 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:36077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 03:19:47.148790 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.148559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:36077 (local address 127.0.0.1:59556)
0929 03:19:47.148692 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:47.148738 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:36077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260929 03:19:47.148922 4150 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.188:36077: Network error: Client connection negotiation failed: client connection to 127.13.238.188:36077: connect: Connection refused (error 111)
I20260929 03:19:47.180091 4150 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } attempt: 1
I20260929 03:19:47.180972 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.180658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:36077 (local address 127.0.0.1:59560)
0929 03:19:47.180811 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:47.180882 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:36077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 03:19:47.181734 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.181503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:36077 (local address 127.0.0.1:59574)
0929 03:19:47.181617 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:47.181669 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:36077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 03:19:47.181865 4150 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.188:36077: Network error: Client connection negotiation failed: client connection to 127.13.238.188:36077: connect: Connection refused (error 111)
I20260929 03:19:47.188714 4225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } attempt: 1
I20260929 03:19:47.189497 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.189228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:36077 (local address 127.0.0.1:59586)
0929 03:19:47.189363 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:47.189422 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:36077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 03:19:47.190313 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.190031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:36077 (local address 127.0.0.1:59596)
0929 03:19:47.190176 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:47.190250 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:36077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260929 03:19:47.190459 4225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.188:36077: Network error: Client connection negotiation failed: client connection to 127.13.238.188:36077: connect: Connection refused (error 111)
W20260929 03:19:47.250563 4232 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:19:47.250926 4232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:47.251001 4232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:47.251065 4232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:47.255215 4232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:47.255273 4232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:47.255298 4232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:47.255321 4232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:47.255350 4232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:47.257053 4150 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } attempt: 2
I20260929 03:19:47.257901 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.257592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:36077 (local address 127.0.0.1:59598)
0929 03:19:47.257753 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:47.257820 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:36077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 03:19:47.258602 4225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } attempt: 2
I20260929 03:19:47.258721 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.258477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:36077 (local address 127.0.0.1:59610)
0929 03:19:47.258596 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:47.258646 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:36077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 03:19:47.258857 4150 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.188:36077: Network error: Client connection negotiation failed: client connection to 127.13.238.188:36077: connect: Connection refused (error 111)
I20260929 03:19:47.259232 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.258975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:36077 (local address 127.0.0.1:59616)
0929 03:19:47.259111 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:47.259165 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:36077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 03:19:47.259794 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.259622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:36077 (local address 127.0.0.1:59620)
0929 03:19:47.259702 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:47.259742 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:36077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 03:19:47.259902 4225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.188:36077: Network error: Client connection negotiation failed: client connection to 127.13.238.188:36077: connect: Connection refused (error 111)
I20260929 03:19:47.261315 4232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:46793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:42551,127.13.238.189:39667,127.13.238.188:36077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.188.keytab
--principal=kudu/127.13.238.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.188:36077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.238.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:47.262473 4232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:47.263532 4232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:47.271926 4238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:47.271930 4240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:47.272142 4237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:47.272253 4232 server_base.cc:1034] running on GCE node
I20260929 03:19:47.272928 4232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:47.274003 4232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:47.275164 4232 hybrid_clock.cc:648] HybridClock initialized: now 1790651987275142 us; error 64 us; skew 500 ppm
Sep 29 03:19:47 dist-test-slave-nbjg krb5kdc[4069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651987, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:47.277777 4232 init.cc:377] Logged in from keytab as kudu/127.13.238.188@KRBTEST.COM (short username kudu)
I20260929 03:19:47.278692 4232 webserver.cc:460] Webserver started at http://127.13.238.188:36805/ using document root <none> and password file <none>
I20260929 03:19:47.279299 4232 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:47.279361 4232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:47.279534 4232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:47.281553 4232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-2/data/instance:
uuid: "b14a3ca2c81548e4b46bda84f719c20c"
format_stamp: "Formatted at 2026-09-29 03:19:47 on dist-test-slave-nbjg"
server_key: "3f9e0819ac7c58409d62aab61ef8bec2"
server_key_iv: "4deabce35167ef6b7e5213ded5b166f4"
server_key_version: "encryptionkey@0"
I20260929 03:19:47.282068 4232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-2/wal/instance:
uuid: "b14a3ca2c81548e4b46bda84f719c20c"
format_stamp: "Formatted at 2026-09-29 03:19:47 on dist-test-slave-nbjg"
server_key: "3f9e0819ac7c58409d62aab61ef8bec2"
server_key_iv: "4deabce35167ef6b7e5213ded5b166f4"
server_key_version: "encryptionkey@0"
I20260929 03:19:47.285704 4232 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:19:47.288182 4247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:47.289412 4232 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 03:19:47.289510 4232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-2/wal
uuid: "b14a3ca2c81548e4b46bda84f719c20c"
format_stamp: "Formatted at 2026-09-29 03:19:47 on dist-test-slave-nbjg"
server_key: "3f9e0819ac7c58409d62aab61ef8bec2"
server_key_iv: "4deabce35167ef6b7e5213ded5b166f4"
server_key_version: "encryptionkey@0"
I20260929 03:19:47.289597 4232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:47.306388 4232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:47.309335 4232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:47.309478 4232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:47.315927 4232 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.188:36077
I20260929 03:19:47.316167 4299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.188:36077 every 8 connection(s)
I20260929 03:19:47.316841 4232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-2/data/info.pb
I20260929 03:19:47.319617 4300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:47.323345 4300 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42551 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:19:47.323712 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 4232
I20260929 03:19:47.323835 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790651713532531-14266-0/minicluster-data/master-2/wal/instance
I20260929 03:19:47.324091 14266 external_mini_cluster.cc:1428] Setting key 15b422338656726ab748809c34d294e8
Sep 29 03:19:47 dist-test-slave-nbjg krb5kdc[4069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651987, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.188@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
Sep 29 03:19:47 dist-test-slave-nbjg krb5kdc[4069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.188@KRBTEST.COM
I20260929 03:19:47.336706 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.325207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40878 (local address 127.13.238.190:42551)
0929 03:19:47.325355 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:47.325360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:47.325940 (+ 580us) server_negotiation.cc:407] Connection header received
0929 03:19:47.326677 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:47.326680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:47.326730 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:47.326821 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:47.327858 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.328645 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.329981 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.330329 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.330435 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:47.332993 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:47.332997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:47.332999 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:47.333024 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:47.334499 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.335095 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.335097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.335099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.335165 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.335474 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.335478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.335480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.335718 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:47.335811 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:47.336382 (+ 571us) server_negotiation.cc:299] Negotiation successful
0929 03:19:47.336514 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:19:47.337821 4303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.325343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59634 (local address 127.13.238.188:36077)
0929 03:19:47.325740 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:47.325754 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:47.325785 (+ 31us) server_negotiation.cc:407] Connection header received
0929 03:19:47.326482 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:47.326501 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:47.326723 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:47.326889 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:47.327752 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.328924 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.330084 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.330621 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.330701 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:47.333336 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:47.333342 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:47.333352 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:47.333373 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:47.335020 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.335546 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.335550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.335552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.335617 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.335874 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.335877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.335878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.336210 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:47.336387 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:47.336590 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 03:19:47.336709 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":263,"spinlock_wait_cycles":2432,"thread_start_us":119,"threads_started":1}
I20260929 03:19:47.337821 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.325342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:42551 (local address 127.0.0.1:40878)
0929 03:19:47.325796 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:47.325844 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:47.326487 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:47.326866 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:47.326872 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:47.327370 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:47.327720 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:47.327740 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.328757 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.328760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:47.329860 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:47.329871 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.330477 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.330480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:47.330626 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:47.331180 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:47.331202 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:47.332855 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:47.334602 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:47.334610 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:47.334621 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:47.335012 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:47.335241 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:47.335243 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:47.335244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:47.335371 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:47.335811 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:47.335816 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:47.336124 (+ 308us) client_negotiation.cc:769] Sending connection context
0929 03:19:47.336317 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 03:19:47.336658 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"spinlock_wait_cycles":42368,"thread_start_us":132,"threads_started":1}
I20260929 03:19:47.339207 4300 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.189" port: 39667 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:19:47 dist-test-slave-nbjg krb5kdc[4069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651987, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.188@KRBTEST.COM for kudu/127.13.238.189@KRBTEST.COM
Sep 29 03:19:47 dist-test-slave-nbjg krb5kdc[4069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:19:47.346792 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.339713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:39667 (local address 127.0.0.1:60562)
0929 03:19:47.339819 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:47.339832 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:47.339901 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:47.340189 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:47.340192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:47.340343 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:47.340436 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:47.340442 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.341149 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.341151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:47.341660 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:47.341666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.342102 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.342106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:47.342251 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:47.342660 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:47.342673 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:47.343777 (+ 1104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:47.345341 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:47.345344 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:47.345346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:47.345574 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:47.345821 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:47.345822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:47.345824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:47.345890 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:47.346315 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:47.346318 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:47.346412 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 03:19:47.346512 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 03:19:47.346623 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 03:19:47.346805 4230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.339834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.13.238.189:39667)
0929 03:19:47.339958 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:47.339963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:47.339976 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:19:47.340020 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:47.340023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:47.340068 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:47.340144 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:47.340542 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.341055 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.341773 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.342001 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.342090 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:47.343907 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:47.343910 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:47.343911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:47.343936 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:47.345234 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.345652 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.345655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.345657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.345739 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.345963 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.345966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.345968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.346230 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:47.346318 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:47.346588 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 03:19:47.346693 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:19:47.347536 4300 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:19:47.349165 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.340542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40894 (local address 127.13.238.190:42551)
0929 03:19:47.340650 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:47.340654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:47.341307 (+ 653us) server_negotiation.cc:407] Connection header received
0929 03:19:47.341398 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:47.341401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:47.341442 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:47.341503 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:47.342058 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.342590 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.343302 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.343519 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.343608 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:47.346069 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:47.346072 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:47.346074 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:47.346094 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:47.347657 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.348082 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.348086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.348087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.348176 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.348430 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.348432 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.348433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.348650 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:47.348893 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:47.348955 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 03:19:47.349046 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:19:47.351052 4225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } attempt: 3
Sep 29 03:19:47 dist-test-slave-nbjg krb5kdc[4069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651987, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.188@KRBTEST.COM for kudu/127.13.238.188@KRBTEST.COM
I20260929 03:19:47.355213 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.348019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:36077 (local address 127.0.0.1:59640)
0929 03:19:47.348226 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:47.348239 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:47.348308 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:47.348498 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:47.348500 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:47.348642 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:47.348725 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:47.348731 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.349624 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.349626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:47.350326 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:47.350336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.350742 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.350745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:47.350850 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:47.351278 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:47.351291 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:47.352517 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:47.353970 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:47.353973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:47.353975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:47.354140 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:47.354388 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:47.354391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:47.354393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:47.354462 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:47.354764 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:47.354767 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:47.354841 (+ 74us) client_negotiation.cc:769] Sending connection context
0929 03:19:47.354919 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:19:47.355040 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":17}
I20260929 03:19:47.355221 4303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.348062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59640 (local address 127.13.238.188:36077)
0929 03:19:47.348158 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:47.348162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:47.348291 (+ 129us) server_negotiation.cc:407] Connection header received
0929 03:19:47.348366 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:47.348368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:47.348411 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:47.348475 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:47.348818 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.349525 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.350433 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.350656 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.350735 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:47.352641 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:47.352644 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:47.352647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:47.352671 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:47.353880 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.354237 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.354239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.354241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.354313 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.354531 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.354534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.354536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.354685 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:47.354762 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:47.354969 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 03:19:47.355097 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":34}
Sep 29 03:19:47 dist-test-slave-nbjg krb5kdc[4069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.189@KRBTEST.COM
Sep 29 03:19:47 dist-test-slave-nbjg krb5kdc[4069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651987, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.189@KRBTEST.COM for kudu/127.13.238.188@KRBTEST.COM
I20260929 03:19:47.358002 4150 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } attempt: 3
I20260929 03:19:47.358466 4300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b14a3ca2c81548e4b46bda84f719c20c: Bootstrap starting.
I20260929 03:19:47.360862 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.351422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:36077 (local address 127.0.0.1:59646)
0929 03:19:47.351799 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:47.351812 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:47.351872 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:47.352103 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:47.352106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:47.352266 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:47.352347 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:47.352353 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.353722 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.353725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:47.354492 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:47.354504 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.355134 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.355137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:47.355218 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:47.355760 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:47.355775 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:47.357004 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:47.359367 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:47.359370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:47.359372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:47.359594 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:47.359882 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:47.359885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:47.359887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:47.359973 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:47.360413 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:47.360416 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:47.360503 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 03:19:47.360579 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 03:19:47.360713 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313}
I20260929 03:19:47.361344 4300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b14a3ca2c81548e4b46bda84f719c20c: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:47.361374 4305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.351560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59646 (local address 127.13.238.188:36077)
0929 03:19:47.351881 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:47.351889 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:47.351902 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:19:47.351935 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:47.351937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:47.351976 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:47.352058 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:47.352455 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.353610 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.354627 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.355003 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.355404 (+ 401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:47.357165 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:47.357168 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:47.357170 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:47.357189 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:47.358874 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.359695 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.359698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.359700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.359792 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.360089 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.360092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.360094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.360307 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:47.361011 (+ 704us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:47.361082 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 03:19:47.361184 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":110,"threads_started":1}
I20260929 03:19:47.362058 4300 log.cc:824] T 00000000000000000000000000000000 P b14a3ca2c81548e4b46bda84f719c20c: Log is configured to *not* fsync() on all Append() calls
Sep 29 03:19:47 dist-test-slave-nbjg krb5kdc[4069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651986, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for kudu/127.13.238.188@KRBTEST.COM
I20260929 03:19:47.364142 4300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b14a3ca2c81548e4b46bda84f719c20c: No bootstrap required, opened a new log
I20260929 03:19:47.364718 4225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18a1861f6de748928118624338c0bcf0: Bootstrap starting.
I20260929 03:19:47.366256 4230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.350257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60576 (local address 127.13.238.189:39667)
0929 03:19:47.350361 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:47.350366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:47.350378 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:19:47.350439 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:47.350442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:47.350478 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:47.350540 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:47.351095 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.351729 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.352716 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.353494 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.353583 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:47.363064 (+ 9481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:47.363068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:47.363071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:47.363094 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:47.364579 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.365058 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.365062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.365063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.365146 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.365415 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.365418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.365420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.365629 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:47.365808 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:47.365975 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 03:19:47.366085 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 03:19:47.367355 4225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18a1861f6de748928118624338c0bcf0: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:47.367938 4300 raft_consensus.cc:348] T 00000000000000000000000000000000 P b14a3ca2c81548e4b46bda84f719c20c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7972ed36fa3542cf8f7e7c15680a214e" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42551 } } peers { permanent_uuid: "18a1861f6de748928118624338c0bcf0" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 39667 } } peers { permanent_uuid: "b14a3ca2c81548e4b46bda84f719c20c" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } }
I20260929 03:19:47.368153 4300 raft_consensus.cc:729] T 00000000000000000000000000000000 P b14a3ca2c81548e4b46bda84f719c20c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b14a3ca2c81548e4b46bda84f719c20c, State: Initialized, Role: FOLLOWER
I20260929 03:19:47.368173 4225 log.cc:824] T 00000000000000000000000000000000 P 18a1861f6de748928118624338c0bcf0: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:47.368513 4300 consensus_queue.cc:260] T 00000000000000000000000000000000 P b14a3ca2c81548e4b46bda84f719c20c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7972ed36fa3542cf8f7e7c15680a214e" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42551 } } peers { permanent_uuid: "18a1861f6de748928118624338c0bcf0" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 39667 } } peers { permanent_uuid: "b14a3ca2c81548e4b46bda84f719c20c" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } }
I20260929 03:19:47.369676 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:19:47.369896 4300 sys_catalog.cc:564] T 00000000000000000000000000000000 P b14a3ca2c81548e4b46bda84f719c20c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:47.370410 4308 sys_catalog.cc:455] T 00000000000000000000000000000000 P b14a3ca2c81548e4b46bda84f719c20c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7972ed36fa3542cf8f7e7c15680a214e" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42551 } } peers { permanent_uuid: "18a1861f6de748928118624338c0bcf0" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 39667 } } peers { permanent_uuid: "b14a3ca2c81548e4b46bda84f719c20c" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } } }
I20260929 03:19:47.370519 4308 sys_catalog.cc:458] T 00000000000000000000000000000000 P b14a3ca2c81548e4b46bda84f719c20c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:19:47.370687 4225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18a1861f6de748928118624338c0bcf0: No bootstrap required, opened a new log
I20260929 03:19:47.371032 4124 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7972ed36fa3542cf8f7e7c15680a214e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40894
I20260929 03:19:47.374919 4225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18a1861f6de748928118624338c0bcf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7972ed36fa3542cf8f7e7c15680a214e" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42551 } } peers { permanent_uuid: "18a1861f6de748928118624338c0bcf0" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 39667 } } peers { permanent_uuid: "b14a3ca2c81548e4b46bda84f719c20c" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } }
W20260929 03:19:47.375627 4319 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b14a3ca2c81548e4b46bda84f719c20c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:19:47.375700 4319 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:19:47.375974 4225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18a1861f6de748928118624338c0bcf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18a1861f6de748928118624338c0bcf0, State: Initialized, Role: FOLLOWER
I20260929 03:19:47.373799 4232 master_runner.cc:186] Error getting master registration for 127.13.238.190:42551: OK, instance_id { permanent_uuid: "7972ed36fa3542cf8f7e7c15680a214e" instance_seqno: 1790651986896029 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:19:47.376339 4225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18a1861f6de748928118624338c0bcf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7972ed36fa3542cf8f7e7c15680a214e" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42551 } } peers { permanent_uuid: "18a1861f6de748928118624338c0bcf0" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 39667 } } peers { permanent_uuid: "b14a3ca2c81548e4b46bda84f719c20c" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } }
I20260929 03:19:47.376441 4232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:19:47.376827 4321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18a1861f6de748928118624338c0bcf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7972ed36fa3542cf8f7e7c15680a214e" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42551 } } peers { permanent_uuid: "18a1861f6de748928118624338c0bcf0" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 39667 } } peers { permanent_uuid: "b14a3ca2c81548e4b46bda84f719c20c" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } } }
I20260929 03:19:47.376955 4321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18a1861f6de748928118624338c0bcf0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:19:47.377635 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.358472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:36077 (local address 127.0.0.1:59658)
0929 03:19:47.358644 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:47.358657 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:47.358745 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:47.358981 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:47.358984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:47.359147 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:47.359242 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:47.359249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.359993 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.359996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:47.360548 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:47.360556 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.361080 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.361082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:47.361147 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:47.361654 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:47.361667 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:47.368869 (+ 7202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:47.375952 (+ 7083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:47.375956 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:47.375958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:47.376214 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:47.376556 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:47.376559 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:47.376561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:47.376672 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:47.377173 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:47.377176 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:47.377283 (+ 107us) client_negotiation.cc:769] Sending connection context
0929 03:19:47.377385 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 03:19:47.377515 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260929 03:19:47.377722 4225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18a1861f6de748928118624338c0bcf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:47.378104 4303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.358537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59658 (local address 127.13.238.188:36077)
0929 03:19:47.358635 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:47.358640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:47.358706 (+ 66us) server_negotiation.cc:407] Connection header received
0929 03:19:47.358828 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:47.358830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:47.358869 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:47.358942 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:47.359364 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.359898 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.360654 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.360865 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.360943 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:47.369018 (+ 8075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:47.369022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:47.369025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:47.369049 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:47.375847 (+ 6798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.376314 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.376318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.376320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.376411 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.376782 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.376786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.376788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.377069 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:47.377165 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:47.377914 (+ 749us) server_negotiation.cc:299] Negotiation successful
0929 03:19:47.378006 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:19:47.380511 4150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e: Bootstrap starting.
I20260929 03:19:47.383211 4150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:47.384060 4150 log.cc:824] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:47.386150 4150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e: No bootstrap required, opened a new log
W20260929 03:19:47.386842 4332 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 18a1861f6de748928118624338c0bcf0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:19:47.386960 4332 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:19:47.384514 4157 master_runner.cc:186] Error getting master registration for 127.13.238.190:42551: OK, instance_id { permanent_uuid: "7972ed36fa3542cf8f7e7c15680a214e" instance_seqno: 1790651986896029 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:19:47.388211 4157 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:19:47.389197 4150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7972ed36fa3542cf8f7e7c15680a214e" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42551 } } peers { permanent_uuid: "18a1861f6de748928118624338c0bcf0" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 39667 } } peers { permanent_uuid: "b14a3ca2c81548e4b46bda84f719c20c" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } }
I20260929 03:19:47.389411 4150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7972ed36fa3542cf8f7e7c15680a214e, State: Initialized, Role: FOLLOWER
I20260929 03:19:47.389731 4150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7972ed36fa3542cf8f7e7c15680a214e" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42551 } } peers { permanent_uuid: "18a1861f6de748928118624338c0bcf0" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 39667 } } peers { permanent_uuid: "b14a3ca2c81548e4b46bda84f719c20c" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } }
I20260929 03:19:47.390231 4335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7972ed36fa3542cf8f7e7c15680a214e" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42551 } } peers { permanent_uuid: "18a1861f6de748928118624338c0bcf0" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 39667 } } peers { permanent_uuid: "b14a3ca2c81548e4b46bda84f719c20c" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } } }
I20260929 03:19:47.390466 4335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:19:47.391072 4150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:47.391147 4124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 03:19:47.391235 4124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:47.392488 4124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7972ed36fa3542cf8f7e7c15680a214e" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42551 } } peers { permanent_uuid: "18a1861f6de748928118624338c0bcf0" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 39667 } } peers { permanent_uuid: "b14a3ca2c81548e4b46bda84f719c20c" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } }
W20260929 03:19:47.392820 4103 tablet.cc:2369] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:19:47.393440 4124 leader_election.cc:288] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e [CANDIDATE]: Term 1 election: Requested vote from peers 18a1861f6de748928118624338c0bcf0 (127.13.238.189:39667), b14a3ca2c81548e4b46bda84f719c20c (127.13.238.188:36077)
I20260929 03:19:47.394739 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7972ed36fa3542cf8f7e7c15680a214e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "18a1861f6de748928118624338c0bcf0"
I20260929 03:19:47.394977 4199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18a1861f6de748928118624338c0bcf0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:47.396281 4199 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 18a1861f6de748928118624338c0bcf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7972ed36fa3542cf8f7e7c15680a214e in term 1.
I20260929 03:19:47.398229 4101 leader_election.cc:302] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a1861f6de748928118624338c0bcf0, 7972ed36fa3542cf8f7e7c15680a214e; no voters:
I20260929 03:19:47.398895 4274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7972ed36fa3542cf8f7e7c15680a214e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b14a3ca2c81548e4b46bda84f719c20c"
I20260929 03:19:47.399139 4274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b14a3ca2c81548e4b46bda84f719c20c [term 0 FOLLOWER]: Advancing to term 1
W20260929 03:19:47.399273 4346 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:19:47.399329 4346 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:19:47.400470 4274 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b14a3ca2c81548e4b46bda84f719c20c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7972ed36fa3542cf8f7e7c15680a214e in term 1.
I20260929 03:19:47.406175 4082 master_runner.cc:263] No leader master found from master 7972ed36fa3542cf8f7e7c15680a214e
I20260929 03:19:47.406265 4082 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:19:47.406428 4335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:47.406718 4335 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e [term 1 LEADER]: Becoming Leader. State: Replica: 7972ed36fa3542cf8f7e7c15680a214e, State: Running, Role: LEADER
I20260929 03:19:47.407119 4335 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7972ed36fa3542cf8f7e7c15680a214e" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42551 } } peers { permanent_uuid: "18a1861f6de748928118624338c0bcf0" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 39667 } } peers { permanent_uuid: "b14a3ca2c81548e4b46bda84f719c20c" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } }
I20260929 03:19:47.409168 4335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7972ed36fa3542cf8f7e7c15680a214e. Latest consensus state: current_term: 1 leader_uuid: "7972ed36fa3542cf8f7e7c15680a214e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7972ed36fa3542cf8f7e7c15680a214e" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42551 } } peers { permanent_uuid: "18a1861f6de748928118624338c0bcf0" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 39667 } } peers { permanent_uuid: "b14a3ca2c81548e4b46bda84f719c20c" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } } }
I20260929 03:19:47.409351 4335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:47.412622 4354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:47.415225 4354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:47.421015 4274 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b14a3ca2c81548e4b46bda84f719c20c [term 1 FOLLOWER]: Refusing update from remote peer 7972ed36fa3542cf8f7e7c15680a214e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:19:47.421833 4352 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b14a3ca2c81548e4b46bda84f719c20c" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:19:47.425861 4308 sys_catalog.cc:455] T 00000000000000000000000000000000 P b14a3ca2c81548e4b46bda84f719c20c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7972ed36fa3542cf8f7e7c15680a214e. Latest consensus state: current_term: 1 leader_uuid: "7972ed36fa3542cf8f7e7c15680a214e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7972ed36fa3542cf8f7e7c15680a214e" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42551 } } peers { permanent_uuid: "18a1861f6de748928118624338c0bcf0" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 39667 } } peers { permanent_uuid: "b14a3ca2c81548e4b46bda84f719c20c" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } } }
I20260929 03:19:47.425980 4308 sys_catalog.cc:458] T 00000000000000000000000000000000 P b14a3ca2c81548e4b46bda84f719c20c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:19:47.427135 4199 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 18a1861f6de748928118624338c0bcf0 [term 1 FOLLOWER]: Refusing update from remote peer 7972ed36fa3542cf8f7e7c15680a214e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:19:47.427791 4352 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a1861f6de748928118624338c0bcf0" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 39667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:19:47.431494 4321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18a1861f6de748928118624338c0bcf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7972ed36fa3542cf8f7e7c15680a214e. Latest consensus state: current_term: 1 leader_uuid: "7972ed36fa3542cf8f7e7c15680a214e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7972ed36fa3542cf8f7e7c15680a214e" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42551 } } peers { permanent_uuid: "18a1861f6de748928118624338c0bcf0" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 39667 } } peers { permanent_uuid: "b14a3ca2c81548e4b46bda84f719c20c" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } } }
I20260929 03:19:47.431625 4321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18a1861f6de748928118624338c0bcf0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:19:47.437525 4303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.409975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59668 (local address 127.13.238.188:36077)
0929 03:19:47.410097 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:47.410101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:47.414465 (+ 4364us) server_negotiation.cc:407] Connection header received
0929 03:19:47.414676 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:47.414679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:47.414730 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:47.414784 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:47.416092 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.416809 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.424128 (+ 7319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.424439 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.424513 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:47.431316 (+ 6803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:47.431321 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:47.431323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:47.431352 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:47.432874 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.433364 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.433367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.433369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.433455 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.433740 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.433743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.433745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.433953 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:47.434058 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:47.437246 (+ 3188us) server_negotiation.cc:299] Negotiation successful
0929 03:19:47.437368 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 03:19:47.437655 4352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7972ed36fa3542cf8f7e7c15680a214e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7972ed36fa3542cf8f7e7c15680a214e" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42551 } } peers { permanent_uuid: "18a1861f6de748928118624338c0bcf0" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 39667 } } peers { permanent_uuid: "b14a3ca2c81548e4b46bda84f719c20c" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } } }
I20260929 03:19:47.437769 4352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:47.440101 4321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18a1861f6de748928118624338c0bcf0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7972ed36fa3542cf8f7e7c15680a214e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7972ed36fa3542cf8f7e7c15680a214e" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42551 } } peers { permanent_uuid: "18a1861f6de748928118624338c0bcf0" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 39667 } } peers { permanent_uuid: "b14a3ca2c81548e4b46bda84f719c20c" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } } }
I20260929 03:19:47.440212 4321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18a1861f6de748928118624338c0bcf0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:19:47.440897 4358 mvcc.cc:205] Tried to move back new op lower bound from 7334510540468445184 to 7334510540423798784. Current Snapshot: MvccSnapshot[applied={T|T < 7334510540468445184}]
I20260929 03:19:47.442288 4361 mvcc.cc:205] Tried to move back new op lower bound from 7334510540468445184 to 7334510540423798784. Current Snapshot: MvccSnapshot[applied={T|T < 7334510540468445184}]
I20260929 03:19:47.442973 4335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7972ed36fa3542cf8f7e7c15680a214e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7972ed36fa3542cf8f7e7c15680a214e" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42551 } } peers { permanent_uuid: "18a1861f6de748928118624338c0bcf0" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 39667 } } peers { permanent_uuid: "b14a3ca2c81548e4b46bda84f719c20c" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } } }
I20260929 03:19:47.443081 4335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:47.443995 4308 sys_catalog.cc:455] T 00000000000000000000000000000000 P b14a3ca2c81548e4b46bda84f719c20c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7972ed36fa3542cf8f7e7c15680a214e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7972ed36fa3542cf8f7e7c15680a214e" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 42551 } } peers { permanent_uuid: "18a1861f6de748928118624338c0bcf0" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 39667 } } peers { permanent_uuid: "b14a3ca2c81548e4b46bda84f719c20c" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 36077 } } }
I20260929 03:19:47.444116 4308 sys_catalog.cc:458] T 00000000000000000000000000000000 P b14a3ca2c81548e4b46bda84f719c20c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:19:47.445843 4359 mvcc.cc:205] Tried to move back new op lower bound from 7334510540468445184 to 7334510540423798784. Current Snapshot: MvccSnapshot[applied={T|T < 7334510540468445184}]
I20260929 03:19:47.448608 4230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.410875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60592 (local address 127.13.238.189:39667)
0929 03:19:47.411001 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:47.411006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:47.416007 (+ 5001us) server_negotiation.cc:407] Connection header received
0929 03:19:47.418077 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:47.418081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:47.418132 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:47.418182 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:47.422300 (+ 4118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.423011 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.427690 (+ 4679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.427983 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.428061 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:47.435512 (+ 7451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:47.435516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:47.435519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:47.435547 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:47.437050 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.447228 (+ 10178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.447232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.447234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.447337 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.447647 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.447651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.447652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.447877 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:47.447989 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:47.448343 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 03:19:47.448466 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 03:19:47.450834 4354 catalog_manager.cc:1347] Generated new cluster ID: 47a1ddb59e11492b9bd868e7924a35d0
I20260929 03:19:47.450892 4354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:47.452600 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:47.411669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40900 (local address 127.13.238.190:42551)
0929 03:19:47.411771 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:47.411775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:47.425391 (+ 13616us) server_negotiation.cc:407] Connection header received
0929 03:19:47.425552 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:47.425555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:47.425604 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:47.425654 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:47.428662 (+ 3008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.429362 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.434536 (+ 5174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:47.434848 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:47.434920 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:47.449051 (+ 14131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:47.449056 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:47.449059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:47.449097 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:47.450741 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.451201 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.451205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.451206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.451290 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:47.451768 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:47.451771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:47.451773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:47.451957 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:47.452049 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:47.452358 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 03:19:47.452462 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:19:47.508308 4354 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:47.509689 4354 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:47.529367 4354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7972ed36fa3542cf8f7e7c15680a214e: Generated new TSK 0
I20260929 03:19:47.530452 4354 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:19:47.564324 4230 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:19:47.561018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60608 (local address 127.13.238.189:39667)
0929 03:19:47.561165 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:47.561169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:47.563183 (+ 2014us) server_negotiation.cc:407] Connection header received
0929 03:19:47.563393 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:47.563397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:47.563457 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:47.563516 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:47.564179 (+ 663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60608: BlockingRecv error: recv got EOF from 127.0.0.1:60608 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 03:19:47.566030 4303 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:19:47.561878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59680 (local address 127.13.238.188:36077)
0929 03:19:47.561992 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:47.561997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:47.565043 (+ 3046us) server_negotiation.cc:407] Connection header received
0929 03:19:47.565235 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:47.565239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:47.565290 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:47.565342 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:47.565901 (+ 559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59680: BlockingRecv error: recv got EOF from 127.0.0.1:59680 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 03:19:47.567423 4154 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:19:47.562549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40912 (local address 127.13.238.190:42551)
0929 03:19:47.562680 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:47.562684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:47.566443 (+ 3759us) server_negotiation.cc:407] Connection header received
0929 03:19:47.566629 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:47.566632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:47.566687 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:47.566740 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:47.567301 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40912: BlockingRecv error: recv got EOF from 127.0.0.1:40912 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 03:19:47.575686 4230 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:19:47.570654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60610 (local address 127.13.238.189:39667)
0929 03:19:47.570777 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:47.570781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:47.574512 (+ 3731us) server_negotiation.cc:407] Connection header received
0929 03:19:47.574824 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:47.574828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:47.574921 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:47.574981 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:47.575568 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60610: BlockingRecv error: recv got EOF from 127.0.0.1:60610 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 03:19:47.584637 4230 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:19:47.582763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60618 (local address 127.13.238.189:39667)
0929 03:19:47.582887 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:47.582892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:47.583563 (+ 671us) server_negotiation.cc:407] Connection header received
0929 03:19:47.583789 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:47.583792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:47.583853 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:47.583909 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:47.584513 (+ 604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60618: BlockingRecv error: recv got EOF from 127.0.0.1:60618 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 03:19:47.590693 4230 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:19:47.589123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60634 (local address 127.13.238.189:39667)
0929 03:19:47.589231 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:47.589236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:47.589735 (+ 499us) server_negotiation.cc:407] Connection header received
0929 03:19:47.589909 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:47.589912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:47.589960 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:47.590006 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:47.590581 (+ 575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60634: BlockingRecv error: recv got EOF from 127.0.0.1:60634 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 03:19:47.598163 4230 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:19:47.596971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60644 (local address 127.13.238.189:39667)
0929 03:19:47.597117 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:47.597121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:47.597369 (+ 248us) server_negotiation.cc:407] Connection header received
0929 03:19:47.597494 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:47.597496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:47.597543 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:47.597606 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:47.598004 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60644: BlockingRecv error: recv got EOF from 127.0.0.1:60644 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260929 03:19:47.609050 4230 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:19:47.607933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60652 (local address 127.13.238.189:39667)
0929 03:19:47.608046 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:47.608049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:47.608295 (+ 246us) server_negotiation.cc:407] Connection header received
0929 03:19:47.608417 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:47.608420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:47.608468 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:47.608530 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:47.608937 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60652: BlockingRecv error: recv got EOF from 127.0.0.1:60652 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260929 03:19:47.628153 4230 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:19:47.627137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.13.238.189:39667)
0929 03:19:47.627255 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:47.627259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:47.627370 (+ 111us) server_negotiation.cc:407] Connection header received
0929 03:19:47.627489 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:47.627491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:47.627539 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:47.627619 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:47.627995 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60668: BlockingRecv error: recv got EOF from 127.0.0.1:60668 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 03:19:47.663358 4230 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:19:47.662304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60680 (local address 127.13.238.189:39667)
0929 03:19:47.662449 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:47.662453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:47.662511 (+ 58us) server_negotiation.cc:407] Connection header received
0929 03:19:47.662658 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:47.662660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:47.662711 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:47.662782 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:47.663235 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60680: BlockingRecv error: recv got EOF from 127.0.0.1:60680 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 03:19:47.730780 4230 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:19:47.729592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60696 (local address 127.13.238.189:39667)
0929 03:19:47.729746 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:47.729751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:47.729856 (+ 105us) server_negotiation.cc:407] Connection header received
0929 03:19:47.730012 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:47.730015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:47.730064 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:47.730134 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:47.730654 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60696: BlockingRecv error: recv got EOF from 127.0.0.1:60696 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 03:19:47.862035 4230 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:19:47.860852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60702 (local address 127.13.238.189:39667)
0929 03:19:47.860979 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:47.860984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:47.861152 (+ 168us) server_negotiation.cc:407] Connection header received
0929 03:19:47.861338 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:47.861341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:47.861391 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:47.861459 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:47.861912 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60702: BlockingRecv error: recv got EOF from 127.0.0.1:60702 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 03:19:48.121316 4230 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:19:48.120172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60710 (local address 127.13.238.189:39667)
0929 03:19:48.120314 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:48.120320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:48.120463 (+ 143us) server_negotiation.cc:407] Connection header received
0929 03:19:48.120611 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:48.120614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:48.120661 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:48.120730 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:48.121200 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60710: BlockingRecv error: recv got EOF from 127.0.0.1:60710 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 03:19:48.377105 4319 catalog_manager.cc:1259] Loaded cluster ID: 47a1ddb59e11492b9bd868e7924a35d0
I20260929 03:19:48.377192 4319 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b14a3ca2c81548e4b46bda84f719c20c: loading cluster ID for follower catalog manager: success
I20260929 03:19:48.379405 4319 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b14a3ca2c81548e4b46bda84f719c20c: acquiring CA information for follower catalog manager: success
I20260929 03:19:48.380597 4319 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b14a3ca2c81548e4b46bda84f719c20c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:19:48.388237 4332 catalog_manager.cc:1259] Loaded cluster ID: 47a1ddb59e11492b9bd868e7924a35d0
I20260929 03:19:48.388314 4332 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 18a1861f6de748928118624338c0bcf0: loading cluster ID for follower catalog manager: success
I20260929 03:19:48.389945 4332 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 18a1861f6de748928118624338c0bcf0: acquiring CA information for follower catalog manager: success
I20260929 03:19:48.391199 4332 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 18a1861f6de748928118624338c0bcf0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:19:48.639325 4430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:48.635591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60714 (local address 127.13.238.189:39667)
0929 03:19:48.635813 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:48.635818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:48.635836 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:19:48.635952 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:48.635954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:48.636121 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:48.636240 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:19:48.636485 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:48.637090 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:48.637938 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:48.638178 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:48.638292 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:48.638543 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:19:48.639015 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:19:48.639111 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:48.639140 (+ 29us) server_negotiation.cc:299] Negotiation successful
0929 03:19:48.639193 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":64,"threads_started":1}
I20260929 03:19:48.645793 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:48.642289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59696 (local address 127.13.238.188:36077)
0929 03:19:48.642525 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:48.642529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:48.642543 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:19:48.642647 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:48.642649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:48.642762 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:48.642855 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:19:48.643006 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:48.643554 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:48.644386 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:48.644625 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:48.644703 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:48.645020 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:19:48.645452 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:19:48.645545 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:48.645588 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 03:19:48.645638 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":67,"threads_started":1}
I20260929 03:19:48.652338 4430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:48.648497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60722 (local address 127.13.238.189:39667)
0929 03:19:48.648616 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:48.648620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:48.649162 (+ 542us) server_negotiation.cc:407] Connection header received
0929 03:19:48.649320 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:48.649324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:48.649378 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:48.649440 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:19:48.649752 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:48.650351 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:48.651187 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:48.651514 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:48.651605 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:48.651853 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:19:48.651940 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:19:48.652001 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:48.652131 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 03:19:48.652194 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 03:19:48.652457 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:48.648684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59710 (local address 127.13.238.188:36077)
0929 03:19:48.648781 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:48.648784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:48.648932 (+ 148us) server_negotiation.cc:407] Connection header received
0929 03:19:48.649041 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:48.649044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:48.649089 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:48.649147 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:19:48.649355 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:48.650062 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:48.651303 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:48.651607 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:48.651685 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:48.652015 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:19:48.652105 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:19:48.652166 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:48.652297 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 03:19:48.652349 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 03:19:48.653275 4442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:48.648501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40928 (local address 127.13.238.190:42551)
0929 03:19:48.649023 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:48.649028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:48.649041 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:19:48.649137 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:48.649140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:48.649324 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:48.649437 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:19:48.649676 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:48.650392 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:48.651465 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:48.651764 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:48.651886 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:48.652157 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:19:48.652506 (+ 349us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:19:48.652628 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:48.652719 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 03:19:48.653129 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":70,"threads_started":1}
I20260929 03:19:48.659919 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 4082
I20260929 03:19:48.665813 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 4157
I20260929 03:19:48.671455 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 4232
2026-09-29T03:19:48Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3024 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:19:48 dist-test-slave-nbjg krb5kdc[4447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:19:48 dist-test-slave-nbjg krb5kdc[4447](info): set up 2 sockets
Sep 29 03:19:48 dist-test-slave-nbjg krb5kdc[4447](info): commencing operation
krb5kdc: starting...
W20260929 03:19:49.626554 14266 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/krb5kdc/test-user.keytab.
Sep 29 03:19:49 dist-test-slave-nbjg krb5kdc[4447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:19:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:19:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
Entry for principal kudu/127.13.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab.
I20260929 03:19:49.742494 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:36895
--webserver_interface=127.13.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:36895,127.13.238.189:45905,127.13.238.188:41683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:19:49.853636 4460 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:19:49.853955 4460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:49.853978 4460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:49.854017 4460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:49.858108 4460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:49.858170 4460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:49.858211 4460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:49.858235 4460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:49.858255 4460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:49.863606 4460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:36895,127.13.238.189:45905,127.13.238.188:41683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.190.keytab
--principal=kudu/127.13.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.190:36895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:49.864650 4460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:49.865643 4460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:19:49.874094 4466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:49.874115 4465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:49.874069 4468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:49.874245 4460 server_base.cc:1034] running on GCE node
I20260929 03:19:49.874960 4460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:49.876104 4460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:49.877270 4460 hybrid_clock.cc:648] HybridClock initialized: now 1790651989877254 us; error 29 us; skew 500 ppm
Sep 29 03:19:49 dist-test-slave-nbjg krb5kdc[4447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651989, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:49.880084 4460 init.cc:377] Logged in from keytab as kudu/127.13.238.190@KRBTEST.COM (short username kudu)
I20260929 03:19:49.880962 4460 webserver.cc:460] Webserver started at http://127.13.238.190:36849/ using document root <none> and password file <none>
I20260929 03:19:49.881500 4460 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:49.881558 4460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:49.881732 4460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:49.883839 4460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-0/data/instance:
uuid: "a7075ec041c54732aa19c4eb65fcfe57"
format_stamp: "Formatted at 2026-09-29 03:19:49 on dist-test-slave-nbjg"
server_key: "de5a832e801619908fda24b27c6410de"
server_key_iv: "11011ef3a433fb71bfc85c31afa18027"
server_key_version: "encryptionkey@0"
I20260929 03:19:49.884358 4460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-0/wal/instance:
uuid: "a7075ec041c54732aa19c4eb65fcfe57"
format_stamp: "Formatted at 2026-09-29 03:19:49 on dist-test-slave-nbjg"
server_key: "de5a832e801619908fda24b27c6410de"
server_key_iv: "11011ef3a433fb71bfc85c31afa18027"
server_key_version: "encryptionkey@0"
I20260929 03:19:49.888007 4460 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 03:19:49.890472 4475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:49.891844 4460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:19:49.891956 4460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-0/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-0/wal
uuid: "a7075ec041c54732aa19c4eb65fcfe57"
format_stamp: "Formatted at 2026-09-29 03:19:49 on dist-test-slave-nbjg"
server_key: "de5a832e801619908fda24b27c6410de"
server_key_iv: "11011ef3a433fb71bfc85c31afa18027"
server_key_version: "encryptionkey@0"
I20260929 03:19:49.892052 4460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:49.904166 4460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:49.907138 4460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:49.907305 4460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:49.913739 4460 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.190:36895
I20260929 03:19:49.913975 4527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.190:36895 every 8 connection(s)
I20260929 03:19:49.914744 4460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-0/data/info.pb
I20260929 03:19:49.917519 4528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:49.918643 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 4460
I20260929 03:19:49.918766 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-0/wal/instance
I20260929 03:19:49.919071 14266 external_mini_cluster.cc:1428] Setting key f470a904aa3c33baa5f00e98564e3af4
I20260929 03:19:49.921921 4528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36895 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.238.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.189@KRBTEST.COM" created.
Sep 29 03:19:49 dist-test-slave-nbjg krb5kdc[4447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651989, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:19:49.936515 4531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:49.924345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35486 (local address 127.13.238.190:36895)
0929 03:19:49.924764 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:49.924782 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:49.925018 (+ 236us) server_negotiation.cc:407] Connection header received
0929 03:19:49.926031 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:49.926036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:49.926269 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:49.926431 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:49.927239 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:49.928085 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:49.929016 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:49.929440 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:49.929609 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:49.931956 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:49.931962 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:49.931971 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:49.931992 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:49.933545 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:49.933987 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:49.933993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:49.933995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:49.934092 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:49.934422 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:49.934425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:49.934427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:49.934792 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:49.934910 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:49.935248 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 03:19:49.935490 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":280,"thread_start_us":114,"threads_started":1}
I20260929 03:19:49.936515 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:49.924346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:36895 (local address 127.0.0.1:35486)
0929 03:19:49.924887 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:49.924922 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:49.925733 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:49.926441 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:49.926446 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:49.926854 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:49.927102 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:49.927117 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:49.928199 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:49.928202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:49.928910 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:49.928919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:49.929603 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:49.929607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:49.929696 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:49.930259 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:49.930278 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:49.931818 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:49.933646 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:49.933655 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:49.933665 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:49.933888 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:49.934223 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:49.934227 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:49.934229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:49.934339 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:49.934894 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:49.934900 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:49.935030 (+ 130us) client_negotiation.cc:769] Sending connection context
0929 03:19:49.935182 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 03:19:49.935549 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":91,"spinlock_wait_cycles":27904,"thread_start_us":153,"threads_started":1}
I20260929 03:19:49.938956 4528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.189" port: 45905 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:19:49.939612 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:49.939373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:45905 (local address 127.0.0.1:43690)
0929 03:19:49.939483 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:49.939548 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.189:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 03:19:49.939800 4476 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.238.189:45905: connect: Connection refused (error 111)
I20260929 03:19:49.940532 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:49.940337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:45905 (local address 127.0.0.1:43692)
0929 03:19:49.940436 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:49.940479 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.189:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 03:19:49.940663 4528 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.189:45905: Network error: Client connection negotiation failed: client connection to 127.13.238.189:45905: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.189.keytab.
Entry for principal kudu/127.13.238.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.189.keytab.
I20260929 03:19:49.943046 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.189:45905
--webserver_interface=127.13.238.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.189.keytab
--principal=kudu/127.13.238.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:36895,127.13.238.189:45905,127.13.238.188:41683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:19:49.989926 4528 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.189" port: 45905 } attempt: 1
I20260929 03:19:49.990859 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:49.990541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:45905 (local address 127.0.0.1:43704)
0929 03:19:49.990700 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:49.990765 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.189:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 03:19:49.991560 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:49.991349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:45905 (local address 127.0.0.1:43720)
0929 03:19:49.991450 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:49.991498 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.189:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 03:19:49.991674 4528 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.189:45905: Network error: Client connection negotiation failed: client connection to 127.13.238.189:45905: connect: Connection refused (error 111)
W20260929 03:19:50.053705 4535 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:19:50.053992 4535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:50.054016 4535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:50.054078 4535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:50.058145 4535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:50.058221 4535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:50.058240 4535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:50.058260 4535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:50.058288 4535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:50.059837 4528 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.189" port: 45905 } attempt: 2
I20260929 03:19:50.060645 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.060350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:45905 (local address 127.0.0.1:43728)
0929 03:19:50.060499 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:50.060564 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.189:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 03:19:50.061412 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.061196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:45905 (local address 127.0.0.1:43742)
0929 03:19:50.061291 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:50.061349 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.189:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 03:19:50.061539 4528 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.189:45905: Network error: Client connection negotiation failed: client connection to 127.13.238.189:45905: connect: Connection refused (error 111)
I20260929 03:19:50.064136 4535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:36895,127.13.238.189:45905,127.13.238.188:41683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.189.keytab
--principal=kudu/127.13.238.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.189:45905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.238.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:50.065274 4535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:50.066356 4535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:19:50.074669 4535 server_base.cc:1034] running on GCE node
W20260929 03:19:50.074608 4540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:50.074610 4543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:50.074608 4541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:50.075351 4535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:50.076478 4535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:50.077654 4535 hybrid_clock.cc:648] HybridClock initialized: now 1790651990077638 us; error 31 us; skew 500 ppm
Sep 29 03:19:50 dist-test-slave-nbjg krb5kdc[4447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651990, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:50.080358 4535 init.cc:377] Logged in from keytab as kudu/127.13.238.189@KRBTEST.COM (short username kudu)
I20260929 03:19:50.081251 4535 webserver.cc:460] Webserver started at http://127.13.238.189:36669/ using document root <none> and password file <none>
I20260929 03:19:50.081781 4535 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:50.081846 4535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:50.082041 4535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:50.084113 4535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-1/data/instance:
uuid: "587842a2454b4412b34c6c63eded8ea6"
format_stamp: "Formatted at 2026-09-29 03:19:50 on dist-test-slave-nbjg"
server_key: "1c62e0314080a334df95b0f3908e9c06"
server_key_iv: "9c6f510f9bc22690d6462c769a9cf808"
server_key_version: "encryptionkey@0"
I20260929 03:19:50.084678 4535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-1/wal/instance:
uuid: "587842a2454b4412b34c6c63eded8ea6"
format_stamp: "Formatted at 2026-09-29 03:19:50 on dist-test-slave-nbjg"
server_key: "1c62e0314080a334df95b0f3908e9c06"
server_key_iv: "9c6f510f9bc22690d6462c769a9cf808"
server_key_version: "encryptionkey@0"
I20260929 03:19:50.088333 4535 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 03:19:50.090739 4550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:50.091965 4535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:19:50.092067 4535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-1/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-1/wal
uuid: "587842a2454b4412b34c6c63eded8ea6"
format_stamp: "Formatted at 2026-09-29 03:19:50 on dist-test-slave-nbjg"
server_key: "1c62e0314080a334df95b0f3908e9c06"
server_key_iv: "9c6f510f9bc22690d6462c769a9cf808"
server_key_version: "encryptionkey@0"
I20260929 03:19:50.092176 4535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:50.108004 4535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:50.110910 4535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:50.111064 4535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:50.117686 4535 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.189:45905
I20260929 03:19:50.117960 4602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.189:45905 every 8 connection(s)
I20260929 03:19:50.118676 4535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-1/data/info.pb
I20260929 03:19:50.118985 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 4535
I20260929 03:19:50.119174 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-1/wal/instance
I20260929 03:19:50.119447 14266 external_mini_cluster.cc:1428] Setting key 3648ca1b6aaa891ef5bf9ad9baa4b62c
I20260929 03:19:50.121430 4603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:50.126531 4603 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36895 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.238.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.238.188@KRBTEST.COM" created.
Sep 29 03:19:50 dist-test-slave-nbjg krb5kdc[4447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651990, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.189@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:19:50.141359 4531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.128837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35500 (local address 127.13.238.190:36895)
0929 03:19:50.128966 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:50.128969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:50.129894 (+ 925us) server_negotiation.cc:407] Connection header received
0929 03:19:50.130944 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:50.130947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:50.131007 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:50.131076 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:50.132248 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.132775 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.134284 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.134535 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.134628 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:50.137232 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:50.137235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:50.137237 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:50.137262 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:50.139238 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.139773 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.139777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.139779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.139855 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.140276 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.140279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.140280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.140475 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:50.140568 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:50.141115 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 03:19:50.141232 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:19:50.142251 4606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.129263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:36895 (local address 127.0.0.1:35500)
0929 03:19:50.129747 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:50.129790 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:50.130721 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:50.131254 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:50.131262 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:50.131597 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:50.132065 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:50.132077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.132893 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.132897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:50.134147 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:50.134158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.134700 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.134704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:50.134903 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:50.135477 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:50.135498 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:50.137056 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:50.139345 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:50.139351 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:50.139368 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:50.139679 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:50.139969 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:50.139972 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:50.139994 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:50.140168 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:50.140580 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:50.140586 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:50.140866 (+ 280us) client_negotiation.cc:769] Sending connection context
0929 03:19:50.141037 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 03:19:50.141304 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":129,"threads_started":1}
I20260929 03:19:50.143761 4603 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.189" port: 45905 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.238.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.188.keytab.
Entry for principal kudu/127.13.238.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.188.keytab.
I20260929 03:19:50.145762 14266 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
/tmp/dist-test-taskdvlSPq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.188:41683
--webserver_interface=127.13.238.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.188.keytab
--principal=kudu/127.13.238.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.238.148:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.238.190:36895,127.13.238.189:45905,127.13.238.188:41683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:19:50 dist-test-slave-nbjg krb5kdc[4447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651990, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.189@KRBTEST.COM for kudu/127.13.238.189@KRBTEST.COM
I20260929 03:19:50.152688 4528 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.189" port: 45905 } attempt: 3
I20260929 03:19:50.153373 4606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.144272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:45905 (local address 127.0.0.1:43754)
0929 03:19:50.144392 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:50.144406 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:50.144488 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:50.145092 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:50.145095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:50.145265 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:50.145371 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:50.145377 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.146657 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.146660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:50.147339 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:50.147349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.147928 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.147932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:50.148023 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:50.148451 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:50.148465 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:50.149730 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:50.151775 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:50.151778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:50.151779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:50.151958 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:50.152243 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:50.152245 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:50.152247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:50.152316 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:50.152896 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:50.152899 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:50.152989 (+ 90us) client_negotiation.cc:769] Sending connection context
0929 03:19:50.153083 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 03:19:50.153217 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 03:19:50.153532 4608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.144359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43754 (local address 127.13.238.189:45905)
0929 03:19:50.144675 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:50.144687 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:50.144711 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:19:50.144754 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:50.144760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:50.144964 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:50.145149 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:50.145479 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.146543 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.147454 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.147825 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.147907 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:50.149862 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:50.149873 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:50.149890 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:50.149916 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:50.151673 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.152062 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.152068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.152070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.152161 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.152392 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.152395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.152397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.152732 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:50.152876 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:50.153234 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 03:19:50.153359 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":103,"threads_started":1}
I20260929 03:19:50.155826 4603 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:19:50.156477 4606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.156240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:41683 (local address 127.0.0.1:57456)
0929 03:19:50.156344 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:50.156416 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:41683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 03:19:50.157087 4554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.238.188:41683: connect: Connection refused (error 111)
I20260929 03:19:50.158258 4606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.158029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:41683 (local address 127.0.0.1:57468)
0929 03:19:50.158120 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:50.158166 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:41683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 03:19:50.158470 4603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.188:41683: Network error: Client connection negotiation failed: client connection to 127.13.238.188:41683: connect: Connection refused (error 111)
Sep 29 03:19:50 dist-test-slave-nbjg krb5kdc[4447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651989, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for kudu/127.13.238.189@KRBTEST.COM
I20260929 03:19:50.162094 4608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.153658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43758 (local address 127.13.238.189:45905)
0929 03:19:50.153918 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:50.153921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:50.153933 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:19:50.153964 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:50.153966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:50.154006 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:50.154088 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:50.154552 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.155261 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.156403 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.156770 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.156869 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:50.159183 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:50.159186 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:50.159188 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:50.159208 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:50.160451 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.160919 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.160922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.160924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.161003 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.161292 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.161295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.161296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.161479 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:50.161561 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:50.161815 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 03:19:50.161915 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":100,"threads_started":1}
I20260929 03:19:50.162091 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.153161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:45905 (local address 127.0.0.1:43758)
0929 03:19:50.153375 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:50.153388 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:50.153468 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:50.154125 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:50.154129 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:50.154338 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:50.154448 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:50.154455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.155408 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.155412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:50.156258 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:50.156271 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.156901 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.156906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:50.157046 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:50.157484 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:50.157500 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:50.159063 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:50.160549 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:50.160552 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:50.160554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:50.160808 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:50.161084 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:50.161087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:50.161089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:50.161171 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:50.161560 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:50.161563 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:50.161678 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 03:19:50.161778 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 03:19:50.161924 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260929 03:19:50.163172 4528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:19:50.163920 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.163676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:41683 (local address 127.0.0.1:57476)
0929 03:19:50.163806 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:50.163857 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:41683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 03:19:50.164776 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.164490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:41683 (local address 127.0.0.1:57478)
0929 03:19:50.164668 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:50.164717 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:41683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260929 03:19:50.164896 4528 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.188:41683: Network error: Client connection negotiation failed: client connection to 127.13.238.188:41683: connect: Connection refused (error 111)
I20260929 03:19:50.196038 4528 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } attempt: 1
I20260929 03:19:50.196992 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.196660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:41683 (local address 127.0.0.1:57486)
0929 03:19:50.196809 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:50.196874 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:41683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 03:19:50.197767 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.197537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:41683 (local address 127.0.0.1:57496)
0929 03:19:50.197658 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:50.197704 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:41683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 03:19:50.197887 4528 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.188:41683: Network error: Client connection negotiation failed: client connection to 127.13.238.188:41683: connect: Connection refused (error 111)
I20260929 03:19:50.207751 4603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } attempt: 1
I20260929 03:19:50.208621 4606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.208309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:41683 (local address 127.0.0.1:57502)
0929 03:19:50.208462 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:50.208539 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:41683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 03:19:50.209424 4606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.209206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:41683 (local address 127.0.0.1:57518)
0929 03:19:50.209319 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:50.209362 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:41683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 03:19:50.209542 4603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.188:41683: Network error: Client connection negotiation failed: client connection to 127.13.238.188:41683: connect: Connection refused (error 111)
W20260929 03:19:50.261683 4609 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:19:50.261976 4609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:19:50.262017 4609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:19:50.262079 4609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:19:50.266093 4609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:19:50.266170 4609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:19:50.266214 4609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:19:50.266237 4609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:19:50.266258 4609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:19:50.271720 4609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.238.148:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.238.190:36895,127.13.238.189:45905,127.13.238.188:41683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/krb5kdc/kudu_127.13.238.188.keytab
--principal=kudu/127.13.238.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.238.188:41683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.238.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 02:25:23 UTC on ccycloud.quasar-imoqlg.root.comops.site
build id 4679193
I20260929 03:19:50.272861 4609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:19:50.273068 4528 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } attempt: 2
I20260929 03:19:50.274013 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.273622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:41683 (local address 127.0.0.1:57522)
0929 03:19:50.273845 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:50.273927 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:41683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 03:19:50.274156 4609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:19:50.274837 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.274604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:41683 (local address 127.0.0.1:57538)
0929 03:19:50.274731 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:50.274776 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:41683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 03:19:50.274951 4528 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.188:41683: Network error: Client connection negotiation failed: client connection to 127.13.238.188:41683: connect: Connection refused (error 111)
I20260929 03:19:50.277689 4603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } attempt: 2
I20260929 03:19:50.278388 4606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.278108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:41683 (local address 127.0.0.1:57544)
0929 03:19:50.278253 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:50.278311 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:41683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 03:19:50.279147 4606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.278897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:41683 (local address 127.0.0.1:57548)
0929 03:19:50.279015 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:50.279074 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.238.188:41683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 03:19:50.279295 4603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.238.188:41683: Network error: Client connection negotiation failed: client connection to 127.13.238.188:41683: connect: Connection refused (error 111)
W20260929 03:19:50.282895 4619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:50.282896 4616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:19:50.282922 4617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:19:50.283027 4609 server_base.cc:1034] running on GCE node
I20260929 03:19:50.283753 4609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:19:50.284924 4609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:19:50.286108 4609 hybrid_clock.cc:648] HybridClock initialized: now 1790651990286071 us; error 45 us; skew 500 ppm
Sep 29 03:19:50 dist-test-slave-nbjg krb5kdc[4447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651990, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:19:50.288865 4609 init.cc:377] Logged in from keytab as kudu/127.13.238.188@KRBTEST.COM (short username kudu)
I20260929 03:19:50.289762 4609 webserver.cc:460] Webserver started at http://127.13.238.188:40677/ using document root <none> and password file <none>
I20260929 03:19:50.290421 4609 fs_manager.cc:359] Metadata directory not provided
I20260929 03:19:50.290491 4609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:19:50.290691 4609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:19:50.292968 4609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-2/data/instance:
uuid: "ca6d502668c24f009aac2af7e85582ef"
format_stamp: "Formatted at 2026-09-29 03:19:50 on dist-test-slave-nbjg"
server_key: "25c552a99f41358ff5de66c5b12f3213"
server_key_iv: "cc31560bbc79c5ce949f109f0248e2c0"
server_key_version: "encryptionkey@0"
I20260929 03:19:50.293570 4609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-2/wal/instance:
uuid: "ca6d502668c24f009aac2af7e85582ef"
format_stamp: "Formatted at 2026-09-29 03:19:50 on dist-test-slave-nbjg"
server_key: "25c552a99f41358ff5de66c5b12f3213"
server_key_iv: "cc31560bbc79c5ce949f109f0248e2c0"
server_key_version: "encryptionkey@0"
I20260929 03:19:50.297657 4609 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:19:50.300133 4626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:19:50.301397 4609 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:19:50.301520 4609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-2/data,/tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-2/wal
uuid: "ca6d502668c24f009aac2af7e85582ef"
format_stamp: "Formatted at 2026-09-29 03:19:50 on dist-test-slave-nbjg"
server_key: "25c552a99f41358ff5de66c5b12f3213"
server_key_iv: "cc31560bbc79c5ce949f109f0248e2c0"
server_key_version: "encryptionkey@0"
I20260929 03:19:50.301620 4609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:19:50.314473 4609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:19:50.317493 4609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:19:50.317667 4609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:19:50.324540 4609 rpc_server.cc:307] RPC server started. Bound to: 127.13.238.188:41683
I20260929 03:19:50.324792 4678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.238.188:41683 every 8 connection(s)
I20260929 03:19:50.325515 4609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-2/data/info.pb
I20260929 03:19:50.328336 4679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:19:50.332144 4679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36895 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:19:50.332617 14266 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu as pid 4609
I20260929 03:19:50.332744 14266 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdvlSPq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790651713532531-14266-0/minicluster-data/master-2/wal/instance
I20260929 03:19:50.332988 14266 external_mini_cluster.cc:1428] Setting key 0fef7883b56b1fa5dff44cef9b051839
Sep 29 03:19:50 dist-test-slave-nbjg krb5kdc[4447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651990, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.188@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
Sep 29 03:19:50 dist-test-slave-nbjg krb5kdc[4447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.188@KRBTEST.COM
I20260929 03:19:50.345163 4531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.334103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35506 (local address 127.13.238.190:36895)
0929 03:19:50.334257 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:50.334262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:50.334956 (+ 694us) server_negotiation.cc:407] Connection header received
0929 03:19:50.335784 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:50.335787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:50.335836 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:50.335907 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:50.336794 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.337344 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.338466 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.338751 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.338832 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:50.341510 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:50.341515 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:50.341517 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:50.341549 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:50.343048 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.343553 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.343557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.343559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.343650 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.344018 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.344021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.344023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.344277 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:50.344349 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:50.344867 (+ 518us) server_negotiation.cc:299] Negotiation successful
0929 03:19:50.344984 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:19:50.345995 4682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.334263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57554 (local address 127.13.238.188:41683)
0929 03:19:50.334714 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:50.334732 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:50.334771 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:19:50.335537 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:50.335575 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:50.335841 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:50.335998 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:50.336710 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.337703 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.338669 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.339136 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.339238 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:50.341867 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:50.341873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:50.341883 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:50.341908 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:50.343595 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.344063 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.344069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.344070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.344140 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.344375 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.344378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.344380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.344611 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:50.344757 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:50.345006 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 03:19:50.345202 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":307,"thread_start_us":130,"threads_started":1}
I20260929 03:19:50.345994 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.334263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.190:36895 (local address 127.0.0.1:35506)
0929 03:19:50.334811 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:50.334855 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:50.335544 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:50.335976 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:50.335981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:50.336404 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:50.336649 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:50.336666 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.337455 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.337458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:50.338350 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:50.338360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.338900 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.338905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:50.339058 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:50.339709 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:50.339730 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:50.341326 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:50.343166 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:50.343174 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:50.343183 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:50.343457 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:50.343755 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:50.343758 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:50.343760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:50.343931 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:50.344374 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:50.344381 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:50.344630 (+ 249us) client_negotiation.cc:769] Sending connection context
0929 03:19:50.344796 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 03:19:50.345123 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"spinlock_wait_cycles":56064,"thread_start_us":152,"threads_started":1}
I20260929 03:19:50.347231 4679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.189" port: 45905 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:19:50 dist-test-slave-nbjg krb5kdc[4447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651990, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.188@KRBTEST.COM for kudu/127.13.238.189@KRBTEST.COM
Sep 29 03:19:50 dist-test-slave-nbjg krb5kdc[4447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.190@KRBTEST.COM
I20260929 03:19:50.355185 4608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.347872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43760 (local address 127.13.238.189:45905)
0929 03:19:50.348005 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:50.348009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:50.348022 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:19:50.348060 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:50.348063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:50.348102 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:50.348168 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:50.348579 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.349098 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.349879 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.350094 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.350177 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:50.352267 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:50.352271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:50.352274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:50.352304 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:50.353637 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.354065 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.354068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.354069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.354145 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.354444 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.354446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.354448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.354610 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:50.354707 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:50.354945 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 03:19:50.355055 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:19:50.355180 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.347733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.189:45905 (local address 127.0.0.1:43760)
0929 03:19:50.347875 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:50.347886 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:50.347952 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:50.348191 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:50.348194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:50.348355 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:50.348464 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:50.348470 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.349237 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.349240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:50.349774 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:50.349782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.350190 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.350221 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:50.350312 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:50.350725 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:50.350739 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:50.352091 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:50.353747 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:50.353750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:50.353752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:50.353977 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:50.354273 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:50.354276 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:50.354279 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:50.354366 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:50.354711 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:50.354715 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:50.354801 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 03:19:50.354885 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 03:19:50.355006 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 03:19:50.355806 4679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:19:50.356143 4531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.348367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35516 (local address 127.13.238.190:36895)
0929 03:19:50.348473 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:50.348477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:50.348489 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:19:50.348559 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:50.348562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:50.348598 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:50.348657 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:50.349196 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.349661 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.350396 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.350610 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.350691 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:50.353090 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:50.353092 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:50.353094 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:50.353116 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:50.354525 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.354877 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.354880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.354882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.354949 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.355180 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.355191 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.355193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.355406 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:50.355609 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:50.355896 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 03:19:50.355998 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Sep 29 03:19:50 dist-test-slave-nbjg krb5kdc[4447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651990, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.188@KRBTEST.COM for kudu/127.13.238.188@KRBTEST.COM
Sep 29 03:19:50 dist-test-slave-nbjg krb5kdc[4447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.238.189@KRBTEST.COM
I20260929 03:19:50.364409 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.356248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:41683 (local address 127.0.0.1:57568)
0929 03:19:50.356508 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:50.356522 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:50.356603 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:50.356860 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:50.356863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:50.357021 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:50.357104 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:50.357109 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.357933 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.357936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:50.358734 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:50.358744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.359306 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.359308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:50.359364 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:50.359759 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:50.359773 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:50.361119 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:50.363157 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:50.363160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:50.363162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:50.363368 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:50.363602 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:50.363605 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:50.363607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:50.363700 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:50.364010 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:50.364013 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:50.364100 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 03:19:50.364178 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:19:50.364268 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
I20260929 03:19:50.364508 4682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.356307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57568 (local address 127.13.238.188:41683)
0929 03:19:50.356405 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:50.356409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:50.356578 (+ 169us) server_negotiation.cc:407] Connection header received
0929 03:19:50.356666 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:50.356667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:50.356705 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:50.356775 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:50.357206 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.357848 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.358859 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.359204 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.359311 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:50.361227 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:50.361230 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:50.361233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:50.361254 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:50.363051 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.363454 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.363456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.363458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.363525 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.363781 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.363782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.363784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.363930 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:50.364000 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:50.364256 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 03:19:50.364362 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:19:50.365468 4608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.357237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43772 (local address 127.13.238.189:45905)
0929 03:19:50.357353 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:50.357357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:50.357402 (+ 45us) server_negotiation.cc:407] Connection header received
0929 03:19:50.357489 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:50.357492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:50.357530 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:50.357599 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:50.358082 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.358828 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.359815 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.360129 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.360228 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:50.362569 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:50.362572 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:50.362575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:50.362600 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:50.363954 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.364369 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.364373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.364374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.364453 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.364727 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.364731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.364733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.364895 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:50.365131 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:50.365228 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 03:19:50.365327 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:19:50.368109 4679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca6d502668c24f009aac2af7e85582ef: Bootstrap starting.
I20260929 03:19:50.368676 14266 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:19:50.370456 4603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } attempt: 3
I20260929 03:19:50.370576 4679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca6d502668c24f009aac2af7e85582ef: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:50.369779 4502 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a7075ec041c54732aa19c4eb65fcfe57"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35516
I20260929 03:19:50.371492 4679 log.cc:824] T 00000000000000000000000000000000 P ca6d502668c24f009aac2af7e85582ef: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:50.373623 4679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca6d502668c24f009aac2af7e85582ef: No bootstrap required, opened a new log
I20260929 03:19:50.374097 4528 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } attempt: 3
Sep 29 03:19:50 dist-test-slave-nbjg krb5kdc[4447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651990, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.189@KRBTEST.COM for kudu/127.13.238.188@KRBTEST.COM
I20260929 03:19:50.378523 4679 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca6d502668c24f009aac2af7e85582ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7075ec041c54732aa19c4eb65fcfe57" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36895 } } peers { permanent_uuid: "587842a2454b4412b34c6c63eded8ea6" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 45905 } } peers { permanent_uuid: "ca6d502668c24f009aac2af7e85582ef" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } }
I20260929 03:19:50.378868 4679 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca6d502668c24f009aac2af7e85582ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca6d502668c24f009aac2af7e85582ef, State: Initialized, Role: FOLLOWER
I20260929 03:19:50.379287 4679 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca6d502668c24f009aac2af7e85582ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7075ec041c54732aa19c4eb65fcfe57" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36895 } } peers { permanent_uuid: "587842a2454b4412b34c6c63eded8ea6" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 45905 } } peers { permanent_uuid: "ca6d502668c24f009aac2af7e85582ef" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } }
I20260929 03:19:50.379815 4686 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6d502668c24f009aac2af7e85582ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7075ec041c54732aa19c4eb65fcfe57" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36895 } } peers { permanent_uuid: "587842a2454b4412b34c6c63eded8ea6" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 45905 } } peers { permanent_uuid: "ca6d502668c24f009aac2af7e85582ef" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } } }
I20260929 03:19:50.379936 4686 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6d502668c24f009aac2af7e85582ef [sys.catalog]: This master's current role is: FOLLOWER
Sep 29 03:19:50 dist-test-slave-nbjg krb5kdc[4447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651989, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.238.190@KRBTEST.COM for kudu/127.13.238.188@KRBTEST.COM
I20260929 03:19:50.380052 4606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.370844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:41683 (local address 127.0.0.1:57580)
0929 03:19:50.370971 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:50.370988 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:50.371072 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:50.371399 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:50.371403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:50.371587 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:50.371699 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:50.371705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.372496 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.372500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:50.373344 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:50.373354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.373867 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.373871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:50.373961 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:50.374506 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:50.374521 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:50.376000 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:50.377991 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:50.377994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:50.377996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:50.378295 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:50.378711 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:50.378714 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:50.378716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:50.378821 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:50.379496 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:50.379500 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:50.379611 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 03:19:50.379724 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 03:19:50.379875 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 03:19:50.380888 4679 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca6d502668c24f009aac2af7e85582ef [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:50.381045 4682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.371057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57580 (local address 127.13.238.188:41683)
0929 03:19:50.371192 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:50.371196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:50.371210 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:19:50.371245 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:50.371247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:50.371291 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:50.371368 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:50.371806 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.372367 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.373461 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.373762 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.373862 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:50.376146 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:50.376150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:50.376153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:50.376198 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:50.377846 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.378385 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.378389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.378391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.378554 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.378980 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.378983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.378985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.379375 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:50.379484 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:50.380675 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0929 03:19:50.380794 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:19:50.384183 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.374560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.238.188:41683 (local address 127.0.0.1:57596)
0929 03:19:50.374690 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 03:19:50.374706 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:19:50.374791 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:19:50.375428 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:19:50.375431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:19:50.375628 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:19:50.375737 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:50.375744 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.376758 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.376762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:50.377822 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:19:50.377835 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.378457 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.378460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:19:50.378541 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:50.378944 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:19:50.378959 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:19:50.380510 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:19:50.382598 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:50.382601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:50.382604 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:50.382870 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:50.383150 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:19:50.383153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:19:50.383155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:19:50.383251 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:19:50.383690 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:19:50.383694 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:19:50.383792 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 03:19:50.383888 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 03:19:50.384025 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 03:19:50.384536 4603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 587842a2454b4412b34c6c63eded8ea6: Bootstrap starting.
I20260929 03:19:50.385177 4685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.374836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57596 (local address 127.13.238.188:41683)
0929 03:19:50.375183 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:50.375192 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:50.375206 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:19:50.375241 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:50.375243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:50.375288 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:50.375372 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:50.375844 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.376651 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.377948 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.378362 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.378452 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:50.380675 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:50.380679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:50.380682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:50.380701 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:50.382500 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.382948 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.382952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.382954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.383040 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.383354 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.383357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.383358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.383552 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:50.383634 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:50.384730 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0929 03:19:50.384847 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":114,"threads_started":1}
W20260929 03:19:50.386739 4697 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca6d502668c24f009aac2af7e85582ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:19:50.386814 4697 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:19:50.387459 4603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 587842a2454b4412b34c6c63eded8ea6: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:50.384836 4609 master_runner.cc:186] Error getting master registration for 127.13.238.190:36895: OK, instance_id { permanent_uuid: "a7075ec041c54732aa19c4eb65fcfe57" instance_seqno: 1790651989911988 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:19:50.388092 4609 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:19:50.388156 4528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57: Bootstrap starting.
I20260929 03:19:50.388371 4603 log.cc:824] T 00000000000000000000000000000000 P 587842a2454b4412b34c6c63eded8ea6: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:50.390650 4603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 587842a2454b4412b34c6c63eded8ea6: No bootstrap required, opened a new log
I20260929 03:19:50.390935 4528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57: Neither blocks nor log segments found. Creating new log.
I20260929 03:19:50.392098 4528 log.cc:824] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57: Log is configured to *not* fsync() on all Append() calls
I20260929 03:19:50.394496 4528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57: No bootstrap required, opened a new log
I20260929 03:19:50.394695 4603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 587842a2454b4412b34c6c63eded8ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7075ec041c54732aa19c4eb65fcfe57" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36895 } } peers { permanent_uuid: "587842a2454b4412b34c6c63eded8ea6" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 45905 } } peers { permanent_uuid: "ca6d502668c24f009aac2af7e85582ef" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } }
I20260929 03:19:50.394941 4603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 587842a2454b4412b34c6c63eded8ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 587842a2454b4412b34c6c63eded8ea6, State: Initialized, Role: FOLLOWER
I20260929 03:19:50.395337 4603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 587842a2454b4412b34c6c63eded8ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7075ec041c54732aa19c4eb65fcfe57" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36895 } } peers { permanent_uuid: "587842a2454b4412b34c6c63eded8ea6" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 45905 } } peers { permanent_uuid: "ca6d502668c24f009aac2af7e85582ef" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } }
I20260929 03:19:50.396163 4701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 587842a2454b4412b34c6c63eded8ea6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7075ec041c54732aa19c4eb65fcfe57" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36895 } } peers { permanent_uuid: "587842a2454b4412b34c6c63eded8ea6" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 45905 } } peers { permanent_uuid: "ca6d502668c24f009aac2af7e85582ef" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } } }
I20260929 03:19:50.396476 4701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 587842a2454b4412b34c6c63eded8ea6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:19:50.397159 4603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 587842a2454b4412b34c6c63eded8ea6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:50.397601 4528 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7075ec041c54732aa19c4eb65fcfe57" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36895 } } peers { permanent_uuid: "587842a2454b4412b34c6c63eded8ea6" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 45905 } } peers { permanent_uuid: "ca6d502668c24f009aac2af7e85582ef" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } }
I20260929 03:19:50.397846 4528 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7075ec041c54732aa19c4eb65fcfe57, State: Initialized, Role: FOLLOWER
I20260929 03:19:50.398247 4528 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7075ec041c54732aa19c4eb65fcfe57" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36895 } } peers { permanent_uuid: "587842a2454b4412b34c6c63eded8ea6" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 45905 } } peers { permanent_uuid: "ca6d502668c24f009aac2af7e85582ef" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } }
I20260929 03:19:50.398810 4707 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7075ec041c54732aa19c4eb65fcfe57" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36895 } } peers { permanent_uuid: "587842a2454b4412b34c6c63eded8ea6" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 45905 } } peers { permanent_uuid: "ca6d502668c24f009aac2af7e85582ef" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } } }
I20260929 03:19:50.399087 4707 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:19:50.399678 4528 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:19:50.400624 4502 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 03:19:50.400746 4502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:50.401993 4502 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7075ec041c54732aa19c4eb65fcfe57" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36895 } } peers { permanent_uuid: "587842a2454b4412b34c6c63eded8ea6" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 45905 } } peers { permanent_uuid: "ca6d502668c24f009aac2af7e85582ef" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } }
W20260929 03:19:50.402599 4713 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 587842a2454b4412b34c6c63eded8ea6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:19:50.402664 4713 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:19:50.402992 4502 leader_election.cc:288] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57 [CANDIDATE]: Term 1 election: Requested vote from peers 587842a2454b4412b34c6c63eded8ea6 (127.13.238.189:45905), ca6d502668c24f009aac2af7e85582ef (127.13.238.188:41683)
I20260929 03:19:50.403715 4653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7075ec041c54732aa19c4eb65fcfe57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ca6d502668c24f009aac2af7e85582ef"
I20260929 03:19:50.403827 4577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7075ec041c54732aa19c4eb65fcfe57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "587842a2454b4412b34c6c63eded8ea6"
I20260929 03:19:50.403949 4653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca6d502668c24f009aac2af7e85582ef [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:50.404042 4577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 587842a2454b4412b34c6c63eded8ea6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:19:50.405261 4653 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ca6d502668c24f009aac2af7e85582ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7075ec041c54732aa19c4eb65fcfe57 in term 1.
I20260929 03:19:50.405396 4577 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 587842a2454b4412b34c6c63eded8ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7075ec041c54732aa19c4eb65fcfe57 in term 1.
I20260929 03:19:50.406776 4476 leader_election.cc:302] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 587842a2454b4412b34c6c63eded8ea6, a7075ec041c54732aa19c4eb65fcfe57; no voters:
W20260929 03:19:50.409484 4725 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:19:50.409559 4725 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:19:50.407614 4535 master_runner.cc:186] Error getting master registration for 127.13.238.190:36895: OK, instance_id { permanent_uuid: "a7075ec041c54732aa19c4eb65fcfe57" instance_seqno: 1790651989911988 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:19:50.410949 4535 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 03:19:50.411129 4481 tablet.cc:2369] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:19:50.416718 4460 master_runner.cc:263] No leader master found from master a7075ec041c54732aa19c4eb65fcfe57
I20260929 03:19:50.416785 4460 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:19:50.416875 4707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:19:50.417114 4707 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57 [term 1 LEADER]: Becoming Leader. State: Replica: a7075ec041c54732aa19c4eb65fcfe57, State: Running, Role: LEADER
I20260929 03:19:50.417450 4707 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7075ec041c54732aa19c4eb65fcfe57" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36895 } } peers { permanent_uuid: "587842a2454b4412b34c6c63eded8ea6" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 45905 } } peers { permanent_uuid: "ca6d502668c24f009aac2af7e85582ef" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } }
I20260929 03:19:50.419334 4707 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7075ec041c54732aa19c4eb65fcfe57. Latest consensus state: current_term: 1 leader_uuid: "a7075ec041c54732aa19c4eb65fcfe57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7075ec041c54732aa19c4eb65fcfe57" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36895 } } peers { permanent_uuid: "587842a2454b4412b34c6c63eded8ea6" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 45905 } } peers { permanent_uuid: "ca6d502668c24f009aac2af7e85582ef" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } } }
I20260929 03:19:50.419438 4707 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:50.426012 4685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.408772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57600 (local address 127.13.238.188:41683)
0929 03:19:50.408877 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:50.408882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:50.410658 (+ 1776us) server_negotiation.cc:407] Connection header received
0929 03:19:50.410990 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:50.410993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:50.411046 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:50.411094 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:50.415070 (+ 3976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.415847 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.418149 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.418516 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.418590 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:50.421148 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:50.421152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:50.421155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:50.421180 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:50.423134 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.424667 (+ 1533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.424670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.424672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.424759 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.425030 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.425033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.425035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.425241 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:50.425372 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:50.425723 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 03:19:50.425847 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:19:50.427850 4608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.409448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43782 (local address 127.13.238.189:45905)
0929 03:19:50.409550 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:50.409553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:50.419211 (+ 9658us) server_negotiation.cc:407] Connection header received
0929 03:19:50.419388 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:50.419391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:50.419442 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:50.419490 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:50.420348 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.421109 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.422255 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.422578 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.422664 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:50.424414 (+ 1750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:50.424418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:50.424421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:50.424450 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:50.426026 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.426527 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.426531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.426533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.426641 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.426938 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.426942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.426944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.427148 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:50.427258 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:50.427581 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 03:19:50.427707 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:19:50.429495 4531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:50.408148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35530 (local address 127.13.238.190:36895)
0929 03:19:50.408254 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:50.408259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:50.410470 (+ 2211us) server_negotiation.cc:407] Connection header received
0929 03:19:50.411286 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:50.411289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:50.411334 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:50.411386 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:50.413082 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.413873 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.417147 (+ 3274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:50.417485 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:50.417554 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:50.420832 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:19:50.420836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:19:50.420838 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:19:50.420866 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:19:50.423986 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.427725 (+ 3739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.427729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.427731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.427818 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:19:50.428514 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:19:50.428518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:19:50.428520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:19:50.428742 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:19:50.428855 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:50.429217 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 03:19:50.429340 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:19:50.434580 4736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:19:50.436801 4736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:19:50.445603 4653 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ca6d502668c24f009aac2af7e85582ef [term 1 FOLLOWER]: Refusing update from remote peer a7075ec041c54732aa19c4eb65fcfe57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:19:50.446301 4734 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca6d502668c24f009aac2af7e85582ef" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:19:50.453140 4686 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6d502668c24f009aac2af7e85582ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7075ec041c54732aa19c4eb65fcfe57. Latest consensus state: current_term: 1 leader_uuid: "a7075ec041c54732aa19c4eb65fcfe57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7075ec041c54732aa19c4eb65fcfe57" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36895 } } peers { permanent_uuid: "587842a2454b4412b34c6c63eded8ea6" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 45905 } } peers { permanent_uuid: "ca6d502668c24f009aac2af7e85582ef" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } } }
I20260929 03:19:50.453265 4686 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6d502668c24f009aac2af7e85582ef [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:19:50.454459 4577 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 587842a2454b4412b34c6c63eded8ea6 [term 1 FOLLOWER]: Refusing update from remote peer a7075ec041c54732aa19c4eb65fcfe57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:19:50.455039 4734 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "587842a2454b4412b34c6c63eded8ea6" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 45905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:19:50.458847 4701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 587842a2454b4412b34c6c63eded8ea6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7075ec041c54732aa19c4eb65fcfe57. Latest consensus state: current_term: 1 leader_uuid: "a7075ec041c54732aa19c4eb65fcfe57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7075ec041c54732aa19c4eb65fcfe57" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36895 } } peers { permanent_uuid: "587842a2454b4412b34c6c63eded8ea6" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 45905 } } peers { permanent_uuid: "ca6d502668c24f009aac2af7e85582ef" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } } }
I20260929 03:19:50.458976 4701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 587842a2454b4412b34c6c63eded8ea6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:19:50.460909 4707 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7075ec041c54732aa19c4eb65fcfe57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7075ec041c54732aa19c4eb65fcfe57" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36895 } } peers { permanent_uuid: "587842a2454b4412b34c6c63eded8ea6" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 45905 } } peers { permanent_uuid: "ca6d502668c24f009aac2af7e85582ef" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } } }
I20260929 03:19:50.461031 4707 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:50.461253 4686 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6d502668c24f009aac2af7e85582ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7075ec041c54732aa19c4eb65fcfe57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7075ec041c54732aa19c4eb65fcfe57" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36895 } } peers { permanent_uuid: "587842a2454b4412b34c6c63eded8ea6" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 45905 } } peers { permanent_uuid: "ca6d502668c24f009aac2af7e85582ef" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } } }
I20260929 03:19:50.461380 4686 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6d502668c24f009aac2af7e85582ef [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:19:50.461445 4737 mvcc.cc:205] Tried to move back new op lower bound from 7334510552849477632 to 7334510552753618944. Current Snapshot: MvccSnapshot[applied={T|T < 7334510552849477632}]
I20260929 03:19:50.463462 4738 mvcc.cc:205] Tried to move back new op lower bound from 7334510552849477632 to 7334510552753618944. Current Snapshot: MvccSnapshot[applied={T|T < 7334510552849477632}]
I20260929 03:19:50.465343 4734 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7075ec041c54732aa19c4eb65fcfe57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7075ec041c54732aa19c4eb65fcfe57" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36895 } } peers { permanent_uuid: "587842a2454b4412b34c6c63eded8ea6" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 45905 } } peers { permanent_uuid: "ca6d502668c24f009aac2af7e85582ef" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } } }
I20260929 03:19:50.465472 4734 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57 [sys.catalog]: This master's current role is: LEADER
I20260929 03:19:50.466568 4701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 587842a2454b4412b34c6c63eded8ea6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7075ec041c54732aa19c4eb65fcfe57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7075ec041c54732aa19c4eb65fcfe57" member_type: VOTER last_known_addr { host: "127.13.238.190" port: 36895 } } peers { permanent_uuid: "587842a2454b4412b34c6c63eded8ea6" member_type: VOTER last_known_addr { host: "127.13.238.189" port: 45905 } } peers { permanent_uuid: "ca6d502668c24f009aac2af7e85582ef" member_type: VOTER last_known_addr { host: "127.13.238.188" port: 41683 } } }
I20260929 03:19:50.466686 4701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 587842a2454b4412b34c6c63eded8ea6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:19:50.467157 4740 mvcc.cc:205] Tried to move back new op lower bound from 7334510552849477632 to 7334510552753618944. Current Snapshot: MvccSnapshot[applied={T|T < 7334510552849477632}]
I20260929 03:19:50.468062 4736 catalog_manager.cc:1347] Generated new cluster ID: e60721f3acb241169cccc2fc7fb41c15
I20260929 03:19:50.468142 4736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:19:50.488387 4736 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:19:50.489682 4736 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:19:50.509923 4736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7075ec041c54732aa19c4eb65fcfe57: Generated new TSK 0
I20260929 03:19:50.510870 4736 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:19:50.545235 4531 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:19:50.542807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35546 (local address 127.13.238.190:36895)
0929 03:19:50.542945 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:50.542951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:50.544147 (+ 1196us) server_negotiation.cc:407] Connection header received
0929 03:19:50.544341 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:50.544345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:50.544396 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:50.544446 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:50.545079 (+ 633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35546: BlockingRecv error: recv got EOF from 127.0.0.1:35546 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 03:19:50.546629 4685 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:19:50.543560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57616 (local address 127.13.238.188:41683)
0929 03:19:50.543712 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:50.543717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:50.545718 (+ 2001us) server_negotiation.cc:407] Connection header received
0929 03:19:50.545861 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:50.545864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:50.545913 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:50.545974 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:50.546510 (+ 536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57616: BlockingRecv error: recv got EOF from 127.0.0.1:57616 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 03:19:50.548653 4608 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:19:50.547222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43784 (local address 127.13.238.189:45905)
0929 03:19:50.547335 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:50.547339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:50.547661 (+ 322us) server_negotiation.cc:407] Connection header received
0929 03:19:50.547821 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:50.547824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:50.547872 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:50.547920 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:50.548535 (+ 615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43784: BlockingRecv error: recv got EOF from 127.0.0.1:43784 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 03:19:50.552588 4608 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:19:50.550993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43794 (local address 127.13.238.189:45905)
0929 03:19:50.551108 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:50.551111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:50.551668 (+ 557us) server_negotiation.cc:407] Connection header received
0929 03:19:50.551873 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:50.551877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:50.551950 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:50.552017 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:50.552494 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43794: BlockingRecv error: recv got EOF from 127.0.0.1:43794 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 03:19:50.557719 4608 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:19:50.556262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43798 (local address 127.13.238.189:45905)
0929 03:19:50.556364 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:50.556368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:50.556809 (+ 441us) server_negotiation.cc:407] Connection header received
0929 03:19:50.556992 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:50.556996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:50.557079 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:50.557130 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:50.557626 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43798: BlockingRecv error: recv got EOF from 127.0.0.1:43798 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 03:19:50.563656 4608 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:19:50.561971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43802 (local address 127.13.238.189:45905)
0929 03:19:50.562083 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:50.562087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:50.562671 (+ 584us) server_negotiation.cc:407] Connection header received
0929 03:19:50.562852 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:50.562856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:50.562906 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:50.562955 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:50.563464 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43802: BlockingRecv error: recv got EOF from 127.0.0.1:43802 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 03:19:50.571660 4608 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:19:50.569989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43816 (local address 127.13.238.189:45905)
0929 03:19:50.570162 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:50.570167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:50.570693 (+ 526us) server_negotiation.cc:407] Connection header received
0929 03:19:50.570885 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:50.570888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:50.570943 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:50.570994 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:50.571536 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43816: BlockingRecv error: recv got EOF from 127.0.0.1:43816 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260929 03:19:50.584229 4608 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:19:50.582320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43822 (local address 127.13.238.189:45905)
0929 03:19:50.582452 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:50.582457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:50.583118 (+ 661us) server_negotiation.cc:407] Connection header received
0929 03:19:50.583354 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:50.583357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:50.583421 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:50.583478 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:50.584105 (+ 627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43822: BlockingRecv error: recv got EOF from 127.0.0.1:43822 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 03:19:50.604444 4608 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:19:50.603064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43838 (local address 127.13.238.189:45905)
0929 03:19:50.603207 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:50.603212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:50.603572 (+ 360us) server_negotiation.cc:407] Connection header received
0929 03:19:50.603714 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:50.603718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:50.603780 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:50.603868 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:50.604319 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43838: BlockingRecv error: recv got EOF from 127.0.0.1:43838 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 03:19:50.639739 4608 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:19:50.638631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43848 (local address 127.13.238.189:45905)
0929 03:19:50.638800 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:50.638805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:50.638866 (+ 61us) server_negotiation.cc:407] Connection header received
0929 03:19:50.639004 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:50.639006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:50.639057 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:50.639131 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:50.639604 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43848: BlockingRecv error: recv got EOF from 127.0.0.1:43848 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260929 03:19:50.707093 4608 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:19:50.705863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43860 (local address 127.13.238.189:45905)
0929 03:19:50.706005 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:50.706010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:50.706221 (+ 211us) server_negotiation.cc:407] Connection header received
0929 03:19:50.706376 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:50.706379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:50.706430 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:50.706498 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:50.706987 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43860: BlockingRecv error: recv got EOF from 127.0.0.1:43860 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 03:19:50.838624 4608 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:19:50.837430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43874 (local address 127.13.238.189:45905)
0929 03:19:50.837603 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:50.837608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:50.837724 (+ 116us) server_negotiation.cc:407] Connection header received
0929 03:19:50.837877 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:50.837880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:50.837930 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:50.838012 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:50.838504 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43874: BlockingRecv error: recv got EOF from 127.0.0.1:43874 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260929 03:19:51.098291 4608 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:19:51.097077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43886 (local address 127.13.238.189:45905)
0929 03:19:51.097233 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:51.097239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:51.097320 (+ 81us) server_negotiation.cc:407] Connection header received
0929 03:19:51.097476 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:51.097479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:51.097559 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:51.097648 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:19:51.098141 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43886: BlockingRecv error: recv got EOF from 127.0.0.1:43886 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 03:19:51.388175 4697 catalog_manager.cc:1259] Loaded cluster ID: e60721f3acb241169cccc2fc7fb41c15
I20260929 03:19:51.388263 4697 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ca6d502668c24f009aac2af7e85582ef: loading cluster ID for follower catalog manager: success
I20260929 03:19:51.389990 4697 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ca6d502668c24f009aac2af7e85582ef: acquiring CA information for follower catalog manager: success
I20260929 03:19:51.391222 4697 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ca6d502668c24f009aac2af7e85582ef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:19:51.403954 4713 catalog_manager.cc:1259] Loaded cluster ID: e60721f3acb241169cccc2fc7fb41c15
I20260929 03:19:51.404030 4713 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 587842a2454b4412b34c6c63eded8ea6: loading cluster ID for follower catalog manager: success
I20260929 03:19:51.405735 4713 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 587842a2454b4412b34c6c63eded8ea6: acquiring CA information for follower catalog manager: success
I20260929 03:19:51.406976 4713 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 587842a2454b4412b34c6c63eded8ea6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:19:51.617033 4808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:51.612662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43890 (local address 127.13.238.189:45905)
0929 03:19:51.612948 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:51.612953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:51.613022 (+ 69us) server_negotiation.cc:407] Connection header received
0929 03:19:51.613200 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:51.613203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:51.613342 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:51.613446 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:19:51.613762 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:51.614349 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:51.615351 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:51.615594 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:51.615694 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:51.616068 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:19:51.616555 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:19:51.616689 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:51.616773 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 03:19:51.616844 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":95,"threads_started":1}
I20260929 03:19:51.624311 4814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:51.620376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57622 (local address 127.13.238.188:41683)
0929 03:19:51.620626 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:51.620629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:51.620645 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:19:51.620864 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:51.620868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:51.620996 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:51.621171 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:19:51.621365 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:51.621893 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:51.622760 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:51.622991 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:51.623079 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:51.623394 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:19:51.623844 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:19:51.623937 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:51.624054 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 03:19:51.624140 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":74,"threads_started":1}
I20260929 03:19:51.631534 4821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:51.627576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35552 (local address 127.13.238.190:36895)
0929 03:19:51.627857 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:51.627862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:51.627877 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:19:51.628009 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:51.628013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:51.628192 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:51.628324 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:19:51.628549 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:51.629260 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:51.630083 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:51.630450 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:51.630540 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:51.630822 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:19:51.631174 (+ 352us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:19:51.631255 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:51.631314 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 03:19:51.631371 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":93,"threads_started":1}
I20260929 03:19:51.631692 4808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:51.627823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43892 (local address 127.13.238.189:45905)
0929 03:19:51.628023 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:51.628028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:51.628046 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:19:51.628197 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:51.628200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:51.628256 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:51.628321 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:19:51.628622 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:51.629298 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:51.630504 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:51.630870 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:51.630987 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:51.631236 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:19:51.631314 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:19:51.631379 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:51.631521 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 03:19:51.631580 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 03:19:51.632346 4814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:19:51.628128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57638 (local address 127.13.238.188:41683)
0929 03:19:51.628270 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 03:19:51.628274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:19:51.628612 (+ 338us) server_negotiation.cc:407] Connection header received
0929 03:19:51.628795 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:19:51.628798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:19:51.628851 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:19:51.628922 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:19:51.629175 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:51.629902 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:51.631100 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:19:51.631382 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:19:51.631472 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:19:51.631893 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:19:51.631976 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:19:51.632027 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 03:19:51.632170 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 03:19:51.632222 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 03:19:51.634747 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 4460
I20260929 03:19:51.640434 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 4535
I20260929 03:19:51.645548 14266 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdvlSPq/build/debug/bin/kudu with pid 4609
2026-09-29T03:19:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2973 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5998 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (278105 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST